CNCF-人工智能工作量的资源和基础设施挑战 RESOURCE AND INFRASTRUCTURE CHALLENGES FOR AI WORKLOADS_第1页
CNCF-人工智能工作量的资源和基础设施挑战 RESOURCE AND INFRASTRUCTURE CHALLENGES FOR AI WORKLOADS_第2页
CNCF-人工智能工作量的资源和基础设施挑战 RESOURCE AND INFRASTRUCTURE CHALLENGES FOR AI WORKLOADS_第3页
CNCF-人工智能工作量的资源和基础设施挑战 RESOURCE AND INFRASTRUCTURE CHALLENGES FOR AI WORKLOADS_第4页
CNCF-人工智能工作量的资源和基础设施挑战 RESOURCE AND INFRASTRUCTURE CHALLENGES FOR AI WORKLOADS_第5页
已阅读5页,还剩16页未读, 继续免费阅读

下载本文档

版权说明:本文档由用户提供并上传,收益归属内容提供方,若内容存在侵权,请进行举报或认领

文档简介

RESOURCEAND

INFRASTRUCTURE

CHALLENGESFOR

AIWORKLOADS

Authors:SabrinaDiLeva,KevinHannon,RajasKakodkar,EkinKarabulut,VictorLu,

AbhishekMalvankar,AlexanderScammon,YuanTang,AndreyVelichkevich,MarlowWarnicke

Contributors:ShravanAchar,AndrewShunichiAikawa,RicardoAravena,MaliniBhandaru,

SachiDesai,JohnuGeorge,JoshHalley,NaadirJeewa,BorisKurktchiev,NikiManoledaki,

ClaudiaMisale,MikeSHNg,RonaldPetty,SudhanshuPrajapati,BrianRedmond,JoelRoberts,VijayRodrigues,KayYan,MichaelYao,AdelZaalouk,CathyZhang

来源:发现报告(

/view?id=5856650

)|引用请注明出处

TABLEOFCONTENTS

SeriesIntroduction03

ExecutiveSummary03

Previously04

ResourceandInfrastructureChallenges04

TopologyAwareness05

ResourceHeterogeneity06

GPUUtilizationandSharing06

Scalability07

I/OBottlenecks07

FaultToleranceandElasticity08

BudgetandCostConstraints09

InfrastructureConsiderations10

GPUDirectCommunications10

NetworkTopologyforAIWorkloads10

What’sNext11

2

CLOUDNATIVE

COMPUTINGFOUNDATION

SeriesIntroduction

Thispaperisthefourthinafive-partseriesonschedulingforcloudnativeAlworkloads:

1.UnderstandingAIWorkloadsforKubernetesScheduling:WhatAlworkloadsareandwhytheydifferfromtraditionalcloud-nativeapplications

2.SchedulingFundamentalsforAIWorkloads:HowKubernetesschedulingworksandwhyadditionallayersareneeded

3.JobOrchestrationChallengesforAIWorkloads:Gangscheduling,fairness,queues,preemption,priority,andreservation

4.ResourceandInfrastructureChallengesforAIWorkloads(thispaper):

Topology,GPUsharing,scalability,I/O,faulttolerance,andcost

5.SolutionsandPracticalGuidanceforAIWorkloadScheduling:Tools,referencearchitectures,andreal-worldusecases

Eachpapercanbereadindependently,buttogethertheyprovideacomprehensiveguidetorunningAlworkloadsonKubernetes.

ExecutiveSummary

Beyondjoborchestration,Alworkloadsfacechallengesrelatedtotheunderlyinginfrastructure:hardwaretopologyaffectscommunicationperformance,

heterogeneousGPUtypesrequirecarefulplacement,expensiveGPUsshouldbesharedwhenpossible,andlong-runningjobsmusttoleratefailures.Thispaperexaminestheseresourceandinfrastructurechallenges:topologyawareness,

resourceheterogeneity,GPUutilizationandsharing,scalability,I/Obottlenecks,faulttoleranceandelasticity,andbudgetconstraints.

4

来源:发现报告(

/view?id=5856650

)|引用请注明出处

Previously

Thefirstthreepapersinthisseriesestablished:

•Paper1:AIworkloadshavefundamentallydifferentcharacteristicsthantraditionalmicroservices.Theyarelong-running,tightlycoupled,GPU-intensive,andtopology-sensitive.

•Paper2:ThedefaultKubernetesscheduleroperatesatthepodlevel.Meta-schedulersandqueuemanagersbridgethegapforAIworkloads.

•Paper3:Joborchestrationchallengesincludegangscheduling,resource

fairness,queuemanagement,preemption,priorityscheduling,andresourcereservation.

Thispaperexamineschallengesrelatedtotheunderlyingresourcesandinfrastructure.

ResourceandInfrastructureChallenges

Thissectionoutlinesschedulingchallengesrelatedtohardwareresourcesand

infrastructure—whereandhowjobsrun.Whilemanyexamplesreferencetrainingworkloads,thesechallengesapplyequallytomulti-nodeinferencedeployments,suchasmodel-parallelordisaggregatedservingarchitectures.

•TopologyAwareness

•ResourceHeterogeneity

•GPUUtilizationandSharing

•Scalability

•I/OBottlenecks

•FaultToleranceandElasticity

•BudgetandCostConstraints

5

来源:发现报告(

/view?id=5856650

)|引用请注明出处

TopologyAwareness

HardwaretopologysignificantlyimpactsAIworkloadperformance.Twodifferentchallengesrequiredifferentsolutions:

•Node-leveltopology:

•NUMA(Non-UniformMemoryAccess).MemoryaccessspeeddependsonwhichCPUsocketisaccessingwhichmemorybank.Placingaworkload’sthreadsandmemoryonthesameNUMAnodeavoidsslowcross-socket

access.

•CPU-GPUaffinity.GPUsareattachedtospecificPCIebuses,whichconnecttospecificCPUsockets.Misalignedplacementaddslatency.

•GPUinterconnectswithinanode.EightGPUsinaservermightbe

connectedviahigh-speedinterconnectsorPCIe(slower).Ajobthatneeds

4GPUsperformsbetterifthose4havehigh-speedconnections.

•Unifiedmemoryarchitectures.Somehardwareplatformsusesystem-on-chip(SoC)designswhereGPUandCPUshareasinglememorypool.Onthesearchitectures,GPUmemoryconsumptiondirectlyreducesavailablesystemmemory.TheKubernetesschedulertreatsGPUmemoryandthememoryavailabletoCPUasindependentresources,unawareofthis

coupling.Thiscanleadtoover-commitmentandsysteminstability(e.g.,OOMkills)whenGPUworkloadsconsumememorythatthescheduler

assumedwasavailableforsystemuse.

•Cluster-leveltopology:

•Networkfabric.Nodesunderthesametop-of-rackswitchcancommunicate

fasterthannodesacrossthedatacenter.Fordistributedtrainingwithfrequentsynchronization,networktopologydominatesperformance.

•Rackplacement.Placingallworkersinthesamerackminimizesnetworkhopsbutcreatesasinglepointoffailure.

•GPUinterconnectsacrossnodes.Technologieslikehigh-bandwidth

networkingwithRDMAprovidehigh-bandwidth,low-latencyconnectionsbetweenGPUsondifferentservers.Theschedulermustknowwhichnodeshavetheseconnections.

•Whythismatters:Forcommunication-heavyworkloadslikedistributed

training,placingworkersonwell-connectednodescanbethedifference

betweenajobtakinghoursvs.days.ThedefaultKubernetesscheduler

haslimitedtopologyawareness—itknowsaboutNUMAbutnotaboutGPUinterconnectsornetworkfabric.

6

来源:发现报告(

/view?id=5856650

)|引用请注明出处

ResourceHeterogeneity

•Problem:Clusterscontainmixedhardware:differentGPUmodels,differentgenerations,differentmemorysizes.SomenodeshaveTPUs,somehave

FPGAs,somehaveonlyCPUs.

•Solutionapproach:

•Nodepoolsgroupnodeswithsimilarhardware.Jobscantargetspecificpools.

•Resourcelabelstagnodeswiththeircapabilities(gpu-type=high-memory,gpu-generation=current).

•Topologyhintshelptheschedulerunderstandhardwarecharacteristicsbeyondsimplelabels.

•Challenges:Jobsmayhavepreferencesratherthanhardrequirements—”preferscurrent-generationGPUsbutcanrunonprevious-generation

withreducedbatchsize.”Expressingandhandlingthesepreferencesaddscomplexity.

GPUUtilizationandSharing

•Problem:GPUsareexpensive,butmanyworkloadsdon’tfullyutilizethem.Aninferenceserviceoraninteractivesessionmightuse20%ofaGPU’scomputecapacity.AllocatingwholeGPUstosuchworkloadswastesresources.

•Solutionapproaches:

•GPUpartitionssupportintohardware-isolatedinstances.AsinglephysicalGPUbecomesmultiplesmallervirtualGPUs,eachwithdedicatedmemoryandcompute.

•Time-slicingsharesaGPUbetweenmultipleworkloadsbyswitching

betweenthemrapidly.Thisimprovesutilizationbutaddslatencyandprovidesnomemoryisolation.

•MPS(Multi-ProcessService)allowsmultiplecontainerstouseasingleGPUconcurrently,ratherthantakingturns.Thisimprovesutilizationforworkloadsthatdon’tneedafullGPUbutprovidesnohardwareisolationbetweenprocesses.

•vGPU(virtualGPU)providessoftware-levelGPUsharing,typicallyinvirtualizedenvironments.

7

来源:发现报告(

/view?id=5856650

)|引用请注明出处

•DynamicResourceAllocation(DRA)isaKubernetesfeaturethatallowsflexibleresourceallocation,includingGPUpartitioningondemandwhichneedstobeimplementedviavendorDRAdrivers.DynamicResource

AllocationreachedGAinKubernetes1.34.

•Caveats:

•Memoryconstraints.LLMsconsumeallavailableGPUmemory.Sharingworksforsmallerworkloads,notforlargemodelinference.

Scalability

•Problem:LargeAItrainingrunsrequiremassivescale—hundredsofthousandsofGPUsacrossthousandsofnodes.Theschedulingsystemmusthandlethis

scale.

•Challenges:

•Clusterautoscaling.ToolslikeKarpenterandtheKubernetesCluster

Autoscaleraddnodeswhendemandexceedscapacity.ButaddingGPUnodestakestime(minutes),andlargejobsmayneedmorenodesthanacloudprovidercanquicklyprovision.

•Workloadautoscaling.Beyondcluster-levelscaling,workloadsneed

toscaledynamically.Elastictraininganddataprocessingjobscan

opportunisticallyexpandbeyondquotawhenresourcesareidleandshrinkbackwhendemandreturns,whilerespectinggangschedulingconstraints.Forinference,scalingisdrivenbyrequestload;metricsliketokens-per-

second,queuedepth,andKVcacheutilizationmaybemorerelevantthanCPUormemory.Disaggregatedinferenceaddscomplexity,asprefillanddecodecomponentsmayscaleindependently.

•Schedulerthroughput.Theschedulermustmakeplacementdecisionsforthousandsofpodswithoutbecomingabottleneck.Atscale,scheduling

overheadmatters—everymillisecondspentperpodaddsup.

•Coordinationoverhead.Gangschedulingatscale(coordinating

placementofthousandsofpodssimultaneously)isharderthanschedulingindependentpods.

I/OBottlenecks

•Problem:GPUsarefast,butgettingdatatothemisslow.I/ObottleneckswasteexpensiveGPUtime.

8

来源:发现报告(

/view?id=5856650

)|引用请注明出处

•Dataloading:ResearchfromGoogleandMicrosoftfoundthatdataloading

consumes10-70%oftrainingexecutiontime.Duringdataloading,GPUssitidle,waitingforthenextbatch.

•Checkpointing:Trainingjobsperiodicallysavemodelstatetostorage.Checkpointingalargemodelcantakeminutesandrequireshighstoragebandwidth.Frequentcheckpointing(forfaulttolerance)vs.infrequent

checkpointing(forperformance)isatradeoff.

•Modelloadingforinference:Beforeservingrequests,inferenceservicesmustloadmodelweightstoGPUmemory.Thetimeandmethodofloadingdependsonthescenario:

•Coldstart:Loadingfromstorage(localfilesystemsorcloudobjectstorage(S3,GCS))cantakeminutesforlargemodels.Optimizedstreamingtools(suchasRun:aiModelStreamer,Tensorizer,andfastsafetensors)reducecoldstarttimebystreamingweightsdirectlyintoGPUmemory.

•Storageperformance:TheMLCommonsStoragebenchmarkmeasures

storagesystemperformanceforMLworkloads.Inadequatestoragethroughputcanbottlenecktheentiretrainingpipeline,regardlessofhowmanyGPUsareavailable.

FaultToleranceandElasticity

•Problem:Long-runningtrainingjobs(daystoweeks)willencounterfailures.Hardwarefails,nodesgodown,preemptionhappens.Withoutfaulttolerance,afailureafter6daysoftrainingmeansstartingover.Thefrequencyofthese

interruptionsissubstantial:Metareportedthatduringa54-daysnapshotofLlama3pre-training,theyexperienced466jobinterruptions:47planned

(firmwareupgrades,configurationupdates)andtheremainderunplannedhardwareorinfrastructurefailures.

•Solutionapproaches:

•Checkpoint/restart:Periodicallysavingmodelstatetodurablestorageallowsresumingfromthelastcheckpointratherthanfromscratch.

Thetradeoff:morefrequentcheckpointsmeanlessworklostbutmoreoverhead.

•Elasticjobs:Someframeworks(likePyTorchElastic)supportchangingthenumberofworkersduringtraining.Ifaworkerfails,trainingcontinueswithfewerworkers(withinlimits).Ifresourcesbecomeavailable,workerscanbeadded.Thisrequires:

•Minimumandmaximumworkercounts

9

来源:发现报告(

/view?id=5856650

)|引用请注明出处

•Abilitytoredistributeworkwhenworkersjoin/leave

•Coordinationwiththeschedulertoadd/removeworkersgracefully

•Role-awarescheduling:theschedulermustunderstandpodroles(e.g.,mastervs.worker)andpreemptworkersbeforemasterstoavoidjob

failure

•Handlingfailureswithoutfullrestart:Forgang-scheduledjobs,one

workerfailuretypicallycrashestheentirejob.Elastictrainingrelaxesthis—thejobcontinueswiththesurvivingworkers,andareplacementworkercanjoinlater.

BudgetandCostConstraints

•Problem:GPUresourcesareexpensive.Teamshavebudgets.Withoutcost

controls,asinglerunawayjobcanconsumeamonth’scloudbudgetovernight.

•Solutionapproaches:

•GPU-hourbudgetslimithowmuchGPUtimeateamorjobcanconsume

•Cost-awareschedulingconsidersthecostofdifferentresourceallocations(spotvs.on-demandinstances,differentGPUtypes)

•Automaticjobterminationwhenbudgetsareexhausted

•Costvisibilityhelpsteamsunderstandandoptimizetheirspending

•Additionally,alongsidefinancialcosts,powerandenergybudgetsshould

alsobeconsideredascriticalinfrastructureconstraintsinenergy-intensiveAIworkloads.Telemetrydata,suchasGPUutilization,powerconsumption,andjobruntimemetrics,canbeleveragedtoenablesmarterschedulingdecisions,optimizeresourceallocation,andenforcebudgetconstraintsmoreeffectively.Thesechallengesremainanareaforfuturedevelopment.

10

来源:发现报告(

/view?id=5856650

)|引用请注明出处

InfrastructureConsiderations

Schedulingdecisionsdependontheunderlyinginfrastructure.ThissectioncoversdatacenterarchitectureandnetworktopologyastheyrelatetoAIworkload

scheduling.

GPUDirectCommunications

GPUscancommunicatewithoutinvolvingtheCPU,whichreduceslatencyandincreasesthroughput.

Withinachassis(sameserver):

•High-speedGPUinterconnectsprovidehigh-bandwidth,low-latency

connectionsbetweenGPUs.InterconnectbandwidthcanexceedhundredsofGB/sbetweenconnectedGPUs,farfasterthanPCIe.

NotallGPUsinallserversarenecessarilyconnectedviahigh-speedlinks.Ina

GPUdenseserveronlycertainpairshaveNVLinkconnections.Theschedulermustunderstandthistopology.

Betweenchassis(acrossservers):

•InfiniBandprovideshigh-bandwidth,low-latencynetworkingbetween

servers,withRDMA(RemoteDirectMemoryAcc

温馨提示

  • 1. 本站所有资源如无特殊说明,都需要本地电脑安装OFFICE2007和PDF阅读器。图纸软件为CAD,CAXA,PROE,UG,SolidWorks等.压缩文件请下载最新的WinRAR软件解压。
  • 2. 本站的文档不包含任何第三方提供的附件图纸等,如果需要附件,请联系上传者。文件的所有权益归上传用户所有。
  • 3. 本站RAR压缩包中若带图纸,网页内容里面会有图纸预览,若没有图纸预览就没有图纸。
  • 4. 未经权益所有人同意不得将文件中的内容挪作商业或盈利用途。
  • 5. 人人文库网仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对用户上传分享的文档内容本身不做任何修改或编辑,并不能对任何下载内容负责。
  • 6. 下载文件中如有侵权或不适当内容,请与我们联系,我们立即纠正。
  • 7. 本站不保证下载资源的准确性、安全性和完整性, 同时也不承担用户因使用这些下载资源对自己和他人造成任何形式的伤害或损失。

评论

0/150

提交评论