版权说明:本文档由用户提供并上传,收益归属内容提供方,若内容存在侵权,请进行举报或认领
文档简介
RESOURCEAND
INFRASTRUCTURE
CHALLENGESFOR
AIWORKLOADS
Authors:SabrinaDiLeva,KevinHannon,RajasKakodkar,EkinKarabulut,VictorLu,
AbhishekMalvankar,AlexanderScammon,YuanTang,AndreyVelichkevich,MarlowWarnicke
Contributors:ShravanAchar,AndrewShunichiAikawa,RicardoAravena,MaliniBhandaru,
SachiDesai,JohnuGeorge,JoshHalley,NaadirJeewa,BorisKurktchiev,NikiManoledaki,
ClaudiaMisale,MikeSHNg,RonaldPetty,SudhanshuPrajapati,BrianRedmond,JoelRoberts,VijayRodrigues,KayYan,MichaelYao,AdelZaalouk,CathyZhang
来源:发现报告(
/view?id=5856650
)|引用请注明出处
TABLEOFCONTENTS
SeriesIntroduction03
ExecutiveSummary03
Previously04
ResourceandInfrastructureChallenges04
TopologyAwareness05
ResourceHeterogeneity06
GPUUtilizationandSharing06
Scalability07
I/OBottlenecks07
FaultToleranceandElasticity08
BudgetandCostConstraints09
InfrastructureConsiderations10
GPUDirectCommunications10
NetworkTopologyforAIWorkloads10
What’sNext11
2
CLOUDNATIVE
COMPUTINGFOUNDATION
SeriesIntroduction
Thispaperisthefourthinafive-partseriesonschedulingforcloudnativeAlworkloads:
1.UnderstandingAIWorkloadsforKubernetesScheduling:WhatAlworkloadsareandwhytheydifferfromtraditionalcloud-nativeapplications
2.SchedulingFundamentalsforAIWorkloads:HowKubernetesschedulingworksandwhyadditionallayersareneeded
3.JobOrchestrationChallengesforAIWorkloads:Gangscheduling,fairness,queues,preemption,priority,andreservation
4.ResourceandInfrastructureChallengesforAIWorkloads(thispaper):
Topology,GPUsharing,scalability,I/O,faulttolerance,andcost
5.SolutionsandPracticalGuidanceforAIWorkloadScheduling:Tools,referencearchitectures,andreal-worldusecases
Eachpapercanbereadindependently,buttogethertheyprovideacomprehensiveguidetorunningAlworkloadsonKubernetes.
ExecutiveSummary
Beyondjoborchestration,Alworkloadsfacechallengesrelatedtotheunderlyinginfrastructure:hardwaretopologyaffectscommunicationperformance,
heterogeneousGPUtypesrequirecarefulplacement,expensiveGPUsshouldbesharedwhenpossible,andlong-runningjobsmusttoleratefailures.Thispaperexaminestheseresourceandinfrastructurechallenges:topologyawareness,
resourceheterogeneity,GPUutilizationandsharing,scalability,I/Obottlenecks,faulttoleranceandelasticity,andbudgetconstraints.
4
来源:发现报告(
/view?id=5856650
)|引用请注明出处
Previously
Thefirstthreepapersinthisseriesestablished:
•Paper1:AIworkloadshavefundamentallydifferentcharacteristicsthantraditionalmicroservices.Theyarelong-running,tightlycoupled,GPU-intensive,andtopology-sensitive.
•Paper2:ThedefaultKubernetesscheduleroperatesatthepodlevel.Meta-schedulersandqueuemanagersbridgethegapforAIworkloads.
•Paper3:Joborchestrationchallengesincludegangscheduling,resource
fairness,queuemanagement,preemption,priorityscheduling,andresourcereservation.
Thispaperexamineschallengesrelatedtotheunderlyingresourcesandinfrastructure.
ResourceandInfrastructureChallenges
Thissectionoutlinesschedulingchallengesrelatedtohardwareresourcesand
infrastructure—whereandhowjobsrun.Whilemanyexamplesreferencetrainingworkloads,thesechallengesapplyequallytomulti-nodeinferencedeployments,suchasmodel-parallelordisaggregatedservingarchitectures.
•TopologyAwareness
•ResourceHeterogeneity
•GPUUtilizationandSharing
•Scalability
•I/OBottlenecks
•FaultToleranceandElasticity
•BudgetandCostConstraints
5
来源:发现报告(
/view?id=5856650
)|引用请注明出处
TopologyAwareness
HardwaretopologysignificantlyimpactsAIworkloadperformance.Twodifferentchallengesrequiredifferentsolutions:
•Node-leveltopology:
•NUMA(Non-UniformMemoryAccess).MemoryaccessspeeddependsonwhichCPUsocketisaccessingwhichmemorybank.Placingaworkload’sthreadsandmemoryonthesameNUMAnodeavoidsslowcross-socket
access.
•CPU-GPUaffinity.GPUsareattachedtospecificPCIebuses,whichconnecttospecificCPUsockets.Misalignedplacementaddslatency.
•GPUinterconnectswithinanode.EightGPUsinaservermightbe
connectedviahigh-speedinterconnectsorPCIe(slower).Ajobthatneeds
4GPUsperformsbetterifthose4havehigh-speedconnections.
•Unifiedmemoryarchitectures.Somehardwareplatformsusesystem-on-chip(SoC)designswhereGPUandCPUshareasinglememorypool.Onthesearchitectures,GPUmemoryconsumptiondirectlyreducesavailablesystemmemory.TheKubernetesschedulertreatsGPUmemoryandthememoryavailabletoCPUasindependentresources,unawareofthis
coupling.Thiscanleadtoover-commitmentandsysteminstability(e.g.,OOMkills)whenGPUworkloadsconsumememorythatthescheduler
assumedwasavailableforsystemuse.
•Cluster-leveltopology:
•Networkfabric.Nodesunderthesametop-of-rackswitchcancommunicate
fasterthannodesacrossthedatacenter.Fordistributedtrainingwithfrequentsynchronization,networktopologydominatesperformance.
•Rackplacement.Placingallworkersinthesamerackminimizesnetworkhopsbutcreatesasinglepointoffailure.
•GPUinterconnectsacrossnodes.Technologieslikehigh-bandwidth
networkingwithRDMAprovidehigh-bandwidth,low-latencyconnectionsbetweenGPUsondifferentservers.Theschedulermustknowwhichnodeshavetheseconnections.
•Whythismatters:Forcommunication-heavyworkloadslikedistributed
training,placingworkersonwell-connectednodescanbethedifference
betweenajobtakinghoursvs.days.ThedefaultKubernetesscheduler
haslimitedtopologyawareness—itknowsaboutNUMAbutnotaboutGPUinterconnectsornetworkfabric.
6
来源:发现报告(
/view?id=5856650
)|引用请注明出处
ResourceHeterogeneity
•Problem:Clusterscontainmixedhardware:differentGPUmodels,differentgenerations,differentmemorysizes.SomenodeshaveTPUs,somehave
FPGAs,somehaveonlyCPUs.
•Solutionapproach:
•Nodepoolsgroupnodeswithsimilarhardware.Jobscantargetspecificpools.
•Resourcelabelstagnodeswiththeircapabilities(gpu-type=high-memory,gpu-generation=current).
•Topologyhintshelptheschedulerunderstandhardwarecharacteristicsbeyondsimplelabels.
•Challenges:Jobsmayhavepreferencesratherthanhardrequirements—”preferscurrent-generationGPUsbutcanrunonprevious-generation
withreducedbatchsize.”Expressingandhandlingthesepreferencesaddscomplexity.
GPUUtilizationandSharing
•Problem:GPUsareexpensive,butmanyworkloadsdon’tfullyutilizethem.Aninferenceserviceoraninteractivesessionmightuse20%ofaGPU’scomputecapacity.AllocatingwholeGPUstosuchworkloadswastesresources.
•Solutionapproaches:
•GPUpartitionssupportintohardware-isolatedinstances.AsinglephysicalGPUbecomesmultiplesmallervirtualGPUs,eachwithdedicatedmemoryandcompute.
•Time-slicingsharesaGPUbetweenmultipleworkloadsbyswitching
betweenthemrapidly.Thisimprovesutilizationbutaddslatencyandprovidesnomemoryisolation.
•MPS(Multi-ProcessService)allowsmultiplecontainerstouseasingleGPUconcurrently,ratherthantakingturns.Thisimprovesutilizationforworkloadsthatdon’tneedafullGPUbutprovidesnohardwareisolationbetweenprocesses.
•vGPU(virtualGPU)providessoftware-levelGPUsharing,typicallyinvirtualizedenvironments.
7
来源:发现报告(
/view?id=5856650
)|引用请注明出处
•DynamicResourceAllocation(DRA)isaKubernetesfeaturethatallowsflexibleresourceallocation,includingGPUpartitioningondemandwhichneedstobeimplementedviavendorDRAdrivers.DynamicResource
AllocationreachedGAinKubernetes1.34.
•Caveats:
•Memoryconstraints.LLMsconsumeallavailableGPUmemory.Sharingworksforsmallerworkloads,notforlargemodelinference.
Scalability
•Problem:LargeAItrainingrunsrequiremassivescale—hundredsofthousandsofGPUsacrossthousandsofnodes.Theschedulingsystemmusthandlethis
scale.
•Challenges:
•Clusterautoscaling.ToolslikeKarpenterandtheKubernetesCluster
Autoscaleraddnodeswhendemandexceedscapacity.ButaddingGPUnodestakestime(minutes),andlargejobsmayneedmorenodesthanacloudprovidercanquicklyprovision.
•Workloadautoscaling.Beyondcluster-levelscaling,workloadsneed
toscaledynamically.Elastictraininganddataprocessingjobscan
opportunisticallyexpandbeyondquotawhenresourcesareidleandshrinkbackwhendemandreturns,whilerespectinggangschedulingconstraints.Forinference,scalingisdrivenbyrequestload;metricsliketokens-per-
second,queuedepth,andKVcacheutilizationmaybemorerelevantthanCPUormemory.Disaggregatedinferenceaddscomplexity,asprefillanddecodecomponentsmayscaleindependently.
•Schedulerthroughput.Theschedulermustmakeplacementdecisionsforthousandsofpodswithoutbecomingabottleneck.Atscale,scheduling
overheadmatters—everymillisecondspentperpodaddsup.
•Coordinationoverhead.Gangschedulingatscale(coordinating
placementofthousandsofpodssimultaneously)isharderthanschedulingindependentpods.
I/OBottlenecks
•Problem:GPUsarefast,butgettingdatatothemisslow.I/ObottleneckswasteexpensiveGPUtime.
8
来源:发现报告(
/view?id=5856650
)|引用请注明出处
•Dataloading:ResearchfromGoogleandMicrosoftfoundthatdataloading
consumes10-70%oftrainingexecutiontime.Duringdataloading,GPUssitidle,waitingforthenextbatch.
•Checkpointing:Trainingjobsperiodicallysavemodelstatetostorage.Checkpointingalargemodelcantakeminutesandrequireshighstoragebandwidth.Frequentcheckpointing(forfaulttolerance)vs.infrequent
checkpointing(forperformance)isatradeoff.
•Modelloadingforinference:Beforeservingrequests,inferenceservicesmustloadmodelweightstoGPUmemory.Thetimeandmethodofloadingdependsonthescenario:
•Coldstart:Loadingfromstorage(localfilesystemsorcloudobjectstorage(S3,GCS))cantakeminutesforlargemodels.Optimizedstreamingtools(suchasRun:aiModelStreamer,Tensorizer,andfastsafetensors)reducecoldstarttimebystreamingweightsdirectlyintoGPUmemory.
•Storageperformance:TheMLCommonsStoragebenchmarkmeasures
storagesystemperformanceforMLworkloads.Inadequatestoragethroughputcanbottlenecktheentiretrainingpipeline,regardlessofhowmanyGPUsareavailable.
FaultToleranceandElasticity
•Problem:Long-runningtrainingjobs(daystoweeks)willencounterfailures.Hardwarefails,nodesgodown,preemptionhappens.Withoutfaulttolerance,afailureafter6daysoftrainingmeansstartingover.Thefrequencyofthese
interruptionsissubstantial:Metareportedthatduringa54-daysnapshotofLlama3pre-training,theyexperienced466jobinterruptions:47planned
(firmwareupgrades,configurationupdates)andtheremainderunplannedhardwareorinfrastructurefailures.
•Solutionapproaches:
•Checkpoint/restart:Periodicallysavingmodelstatetodurablestorageallowsresumingfromthelastcheckpointratherthanfromscratch.
Thetradeoff:morefrequentcheckpointsmeanlessworklostbutmoreoverhead.
•Elasticjobs:Someframeworks(likePyTorchElastic)supportchangingthenumberofworkersduringtraining.Ifaworkerfails,trainingcontinueswithfewerworkers(withinlimits).Ifresourcesbecomeavailable,workerscanbeadded.Thisrequires:
•Minimumandmaximumworkercounts
9
来源:发现报告(
/view?id=5856650
)|引用请注明出处
•Abilitytoredistributeworkwhenworkersjoin/leave
•Coordinationwiththeschedulertoadd/removeworkersgracefully
•Role-awarescheduling:theschedulermustunderstandpodroles(e.g.,mastervs.worker)andpreemptworkersbeforemasterstoavoidjob
failure
•Handlingfailureswithoutfullrestart:Forgang-scheduledjobs,one
workerfailuretypicallycrashestheentirejob.Elastictrainingrelaxesthis—thejobcontinueswiththesurvivingworkers,andareplacementworkercanjoinlater.
BudgetandCostConstraints
•Problem:GPUresourcesareexpensive.Teamshavebudgets.Withoutcost
controls,asinglerunawayjobcanconsumeamonth’scloudbudgetovernight.
•Solutionapproaches:
•GPU-hourbudgetslimithowmuchGPUtimeateamorjobcanconsume
•Cost-awareschedulingconsidersthecostofdifferentresourceallocations(spotvs.on-demandinstances,differentGPUtypes)
•Automaticjobterminationwhenbudgetsareexhausted
•Costvisibilityhelpsteamsunderstandandoptimizetheirspending
•Additionally,alongsidefinancialcosts,powerandenergybudgetsshould
alsobeconsideredascriticalinfrastructureconstraintsinenergy-intensiveAIworkloads.Telemetrydata,suchasGPUutilization,powerconsumption,andjobruntimemetrics,canbeleveragedtoenablesmarterschedulingdecisions,optimizeresourceallocation,andenforcebudgetconstraintsmoreeffectively.Thesechallengesremainanareaforfuturedevelopment.
10
来源:发现报告(
/view?id=5856650
)|引用请注明出处
InfrastructureConsiderations
Schedulingdecisionsdependontheunderlyinginfrastructure.ThissectioncoversdatacenterarchitectureandnetworktopologyastheyrelatetoAIworkload
scheduling.
GPUDirectCommunications
GPUscancommunicatewithoutinvolvingtheCPU,whichreduceslatencyandincreasesthroughput.
Withinachassis(sameserver):
•High-speedGPUinterconnectsprovidehigh-bandwidth,low-latency
connectionsbetweenGPUs.InterconnectbandwidthcanexceedhundredsofGB/sbetweenconnectedGPUs,farfasterthanPCIe.
NotallGPUsinallserversarenecessarilyconnectedviahigh-speedlinks.Ina
GPUdenseserveronlycertainpairshaveNVLinkconnections.Theschedulermustunderstandthistopology.
Betweenchassis(acrossservers):
•InfiniBandprovideshigh-bandwidth,low-latencynetworkingbetween
servers,withRDMA(RemoteDirectMemoryAcc
温馨提示
- 1. 本站所有资源如无特殊说明,都需要本地电脑安装OFFICE2007和PDF阅读器。图纸软件为CAD,CAXA,PROE,UG,SolidWorks等.压缩文件请下载最新的WinRAR软件解压。
- 2. 本站的文档不包含任何第三方提供的附件图纸等,如果需要附件,请联系上传者。文件的所有权益归上传用户所有。
- 3. 本站RAR压缩包中若带图纸,网页内容里面会有图纸预览,若没有图纸预览就没有图纸。
- 4. 未经权益所有人同意不得将文件中的内容挪作商业或盈利用途。
- 5. 人人文库网仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对用户上传分享的文档内容本身不做任何修改或编辑,并不能对任何下载内容负责。
- 6. 下载文件中如有侵权或不适当内容,请与我们联系,我们立即纠正。
- 7. 本站不保证下载资源的准确性、安全性和完整性, 同时也不承担用户因使用这些下载资源对自己和他人造成任何形式的伤害或损失。
最新文档
- 小学五年级劳动课“回味炒菜我会做”教学设计
- 小学五年级道德与法治教学设计:规则内化与自律养成-以“越自律越自由”为例
- 小学少儿小主持三级第一单元第2课《把一句话说亮:停连、重音与对象感》教案
- 新人教版2026-2027学年七年级地理上册《人口和人种》单元教学设计
- 初中八年级数学因式分解复习课教学设计
- 小学六年级道德与法治教案 红领巾相约中国梦主题班会设计
- 小学六年级道德与法治教学设计《守护者的法治密码》
- 七年级地理《矿产资源与工业》教学设计
- (2026版)食堂工作总结
- (2026年)度部门工作总结及计划
- 2026年心肺康复学习题库(含答案)
- DL-T 5220-2021《10kV及以下架空配电线路设计规范》
- 110kV变电站可行性研究报告
- (2026全考点)N1叉车司机证理论模拟考试有答案
- 市政工程施工项目部安全资料目录
- 2026年江苏省普通高中学业水平合格性考试物理学科必修第三册必考知识点清单
- 幼儿园诚信工作制度
- 雨课堂学堂在线学堂云《人工智能时代的创新思维(北京理工)》单元测试考核答案
- 初中采购管理制度
- 2026年内蒙古化工职业学院单招职业技能考试题库附参考答案详解(考试直接用)
- (正式版)DB21∕T 2613-2025 《铁路交通安全运营气象指标》
评论
0/150
提交评论