2024AI异构计算集群调度与资源管理系统

enant)10GPU

提交TF

20GPU

100GPU

提交MXNetGPU本地GPU本地pythontrain.py--batch_size=256--GPU"jobName":"dataDir":"/tmp/data",

"taskRoles":"taskNumber":"taskNumber":"cpuNumber":"memoryMB":"gpuNumber":

"pythontrain.py--batch_size=256

Docker

Docker

DockerunionARGRUNapt-getupdate&&apt-getinstall-y--no-install-recommendsRUNcurl-o/opt/conda/bin/condainstall-y-cpytorchmagma-cuda100&&WORKDIRCOPY.WORKDIR/workspaceRUNchmod-Ra+w.PyorchDockerImagefilefilefilefilefilefile filefile file file fileImagelayerImagelayerImagebaseAUFSDLDLDLDLDLDLDLDLLinuxcontainersofoperatingsystem-virtualizationfortheLinuxoperatingcgroup,

RootPIDPIDPIDNamespacepid2(pidpid4(pidpid5(pidpid3(pidpid1(pidblack:Realblue:getid()togetthis控制组(Control控制计数CGOUPCGOUPCGOUPCGOUPCGOUPCGOUPCGOUPblockConfineprocessestoprocessorandmemorynodeForshowingprivateview(namespace)ofsystemtoprocessesinShareCPUbandwidthbetweenTheCPUAccounting(cpuacct)subsystemgeneratesautomaticreportsonCPUresourcesThememorycontrollersupportsreportingandlimitingofmemory,kernelmemory,andswapusedbyThissupportscontrollingwhichprocessesmaycreate(mknod)devicesaswellasopenthemforreadingorwriting.TheRDMAcontrollerpermitslimitingtheuseofRDMA/IB-resourcesperTheblkiocgroupcontrolsandlimitsaccesstospecifiedblockdevicesbyapplyingIOcontrolGPUNvidiaGPUNvidiaMPSNvidiaDocker#Testnvidia-smiwiththelatestofficialCUDA$dockerrun--gpusallnvidia/cuda:9.0-basenvidia-#StartaGPUenabledcontainerontwo$dockerrun--gpus2nvidia/cuda:9.0-basenvidia-#StartingaGPUenabledcontaineronspecific$dockerrun--gpus'"device=1,2"'nvidia/cuda:9.0-basenvidia-(GangDRF(DominantResourceFairness(Capacity利用率(UtilizationJobA JobB 目标：HighThroughput,HighUtilizationandShortResponseTimes

Node Node 无法启动，无法启动，JobA(Gang

JobA JobB

JobCAschedulingalgorithmforparallelsystemsthatschedulesrelatedthreadsorprocessestorunsimultaneouslyondifferentprocessors.

Node NodeGPUClusterGPUClusterJobJob14GB14GBJob22GB32GBCPU,Hostmemoryetc.)，并且需要调度GPU及GPUmemoryClusterResources:[10GPU,20GBRAMDominantResourceFairness(smallestdominant定主导资源(dominantresource)基于最大最小公平（max-minfairness）的针对多资源类型（e.gGPU,CPU）的调

JobQueueJob14GB1QueueJob14GB14GBJob22GB32GBTotalMemory4+4=8GBMemoryShare=8/20=0.4SHARE=0.4[DominantresourceisJobTotalGPU2+3=5GPUGPUShare=5/10=TotalMemory2+2=4GBMemoryShare=4/20=0.2SHARE=0.5[DominantresourceisClusterResources:[10GPU,20GBJob1hashigherprioritythanJob2asJob1share0.4islessthanJob2share设计目标：Utilization,Fairnessand

UsedCapacityFreeUsedCapacityFreeCapacityUsedCapacityNofreeresources,jobcannotbe(CapacityMinCapacityMaxUserUserLimitFactor=VirtualQueueUserQueueUserLimitFactor=MinCapacityMaxCapacityUserLimitFactor：控制单用户的可以消耗的QueueQueueUserLimitFactor=MinCapacityMaxCapacity虚拟集群(Virtual8-GPU8-GPU

8-GPU

App2时分复用(Timeslicing)与超额订阅迁移Gandiva(early(cluster(cluster-levelTwoReactiveReactive作业到达(arrivals(departures),NodeswithsameNodeswith“noNodeswith“differentOversubscription:suspend-onsame“affinity”JobIntrospective早反馈(EarlytimeGPU软件栈对GPU状态备份，资源隔离做的不够完善，无法很好的支持时分复用(timeslicing)，装箱PAIPAIPAIPAIPAIBigPAIDL利用K8s可扩展DeepManagedbyManagedbyManagedby模块化高效:细粒度GPU调度,鲁棒性云原生

VS(Code),

KubernetesKubernetesClusterDocker/Ope