麦肯锡 -AI 推理规模化 Scaling AI inference

October2026

McKinsey

&Company

SemiconductorsPractice

ScalingAIinference

InferenceisreshapingAleconomics,infrastructurearchitectures,andinvestmentprioritiesacrossthesemiconductorecosystem.

byBillWiseman

withPranjaliKumarandYvonneFerrier

Albuild-outhaspassedathreshold.Foryears,alargeshareofcomputeresourcesweredevotedto

training,drivenbyacceleratorscarcity,frontiermodelraces,andtheneedtoassembleincreasinglylargeclusters-oftenthousandstotensofthousandsofgraphicsprocessingunits(GPUs)-totrainthosemodels.By2030,theAlworkloadmixisexpectedtoshifttowardinference,whichwillaccountforabout60percentofAldemand,comparedwithabout40percentfortraining(Exhibit1).TheindustryisshiftingfrombuildingAlintelligencetoservingitatscale,afundamentallydifferentengineeringchallengewithadistinctcost

structure(seesidebar,"Latency,throughput,andcostconstraintsvaryacrossinferencerequests").

Indeed,C-suiteleadersarenolongeraskingwhetherAlcreatesvaluebutwhetheritdoessoatan

acceptablecosttoserve.EnterprisespayforAluseandinfrastructureimmediately.Processredesign,dataintegration,employeeadoption,andmeasurableproductivitygainstakelonger.AnAldeploymentcanlookveryexpensivebeforeitlookssuccessful,andexecutivesexpectcoststorisemateriallyasautonomous

systemstakeonanincreasingshareofwork;muchofAladoptionandtheinfrastructuredemandrequiredtosupportitmaystilllieahead.Butunlesscosttoservedecreasessignificantly,enterprisesmaynotbeabletoadoptAlinferenceatscale.

Exhibit1

Alworkloads,especiallyinference,accountforthebulkofdatacenterdemand.

Demandbyworkloadtype(continuedmomentumscenario),%

278gigawatts

120

Altraining

Alinference

Non-Al

Alinferencegrowth

+36%

perannum

TotalAlgrowth

+30%

perannum

gigawatts

29

29

42

2026

28

42

30

60%

ofallAl

demand

2030E

McKinsey&Company

Latency,throughput,andcostconstraintsvaryacrossinferencerequests

Trainingtypicallyrunslong-lived,highlyoptimizedjobsonrelativelyhomogeneousacceleratorclus-

ters.Incontrast,inferenceisfundamentallyheterogeneous,bothintermsofhardwareandconstraints;latency,throughput,andcostconstraintsdifferacrossworkloadsanddivergeagainbetweenprefillanddecodewithinasinglerequest.Considerthefollowingkindsofrequests:

一Agenticorcode:longorrepeatedcontext,multistep,andcompute-andKV(key-value)cache-intensive

-Chatandsearch:interactiveandlatency-sensitive

一Voice:streaming,real-time,andlatency-sensitive

一Mediageneration:batch-oriented,throughput-sensitive,andhighcomputeperoutput

-PhysicalAl:ondeviceandconstrainedbylatencyandbypowerandthermalbudgets

Tounderstandthisnewenvironment,McKinseycollaboratedwiththeGlobalSemiconductorAlliance(GSA)toconductinterviewswithAlarchitectsfromleadingsemiconductorplayersandhyperscalers.Fromtheseconversations,we'vedistilledfiveshiftsthatwilldefinetheeraofAlinference:

1.EnterpriseswillpredominantlyjudgeAlperformancebycostperusefuloutcomeratherthancomponentbenchmarks,potentiallymovingvaluetowardthevaluechainlayersthatimprovesystemeconomics.

2.Workloadsarebecomingmorediverse,creatingdemandforspecializedsilicon(whichcanhappenthroughleverssuchas[semi-]customsiliconorchipletdesigns).

3.Computeforinferencewillbedistributedacrosscloud,enterprise,edge,anddevice,withlocationdictatedbyeconomics,latency,anddatacontrol.

4.Hardware-logic,memory,andnetworking-dictateswhatAlcomputecandeliver,whilesoftwaredictateshowmuchofthatcapabilitycanbecaptured.

5.Specializationpaysonlyatscale,makingopenstandardsanecessity.

Below,weexploretheseshiftsindetail,aswellasproposehowthesemiconductorindustryshouldrespondtoaddresstheconcernsofenterpriseleadersandsuccessfullyscaleAlinference.

1.Enterprisesmayweighcostperoutcomeratherthangenericbenchmarks

Thehardwareroadmapsalreadypointtowardmoreperformance,morememory,fasterinterconnect,anddensersystems.Thechallengeisthateachincrementnowcostsmorethanthelast,andthereturnsarenonlinear.SoasenterprisesdecidewheretoinvestinAl,theyneedbetterdemandsignalsthangenericbenchmarksalonecanprovide.

Becauseenterprisedemandisfragmentedbyindustry,Alrequirementsshouldbederivedfromthe

businessoutcome.Aretailrecommendermayneedtoservemillionsofrelativelylow-costrequestswithhighavailabilityandpredictableuniteconomics,whileanautonomousvehiclerequiresnear-real-timeinferencewithstringentlatencyandreliabilityrequirements.Theseworkloadshaveverydifferentrequirements

forinfrastructure,cost,performance,andservicelevels,whichasinglegeneral-purposebenchmarkcannotcapture.

Insteadoflookingatwhetherthespecsheetimproved,enterprisesmustconsiderwhatagivenworkloadcoststoservereliably,includingthecostperresolvedserviceticket,perlineofacceptedcode,andperinspectedunit.Giventhisshift,thecompaniesbuildinghardwareandthecustomersdeployingitmustcollaborateearlyintheprocesswhilearchitecturalchoicescanstillbeshaped.

Thischangeinwhattheindustrymeasureswillikelyalsomovevalue.Today,valuecaptureisheavily

influencedbyscarcity;leadingaccelerators,high-bandwidthmemory(HBM),advancedpackaging,and

accesstopowerallcommandpremiums,partlybecausesupplyhasstruggledtokeeppace.Durablevaluemayaccrueinsteadtothelayersthatmeasurablyimprovesystemeconomics,suchasloweringcost,raisingutilization,cuttinglatency,orincreasingusefuloutputperwatt.Amemorysupplierthatrelievesabandwidthbottleneck,aninterconnectvendorthatkeepsarackusefullycoherent,acoolingorpower-deliveryplayerthatletsdensityrise-eachearnsitspositionbyimprovingwhatthewholesystemdelivers.

Near-termgainswilllikelyremainconcentratedamongmemorysuppliers,leadingacceleratorvendors,

andasmallgroupofcustom-siliconandnetworkingcompanies.Butasconstraintsmovebeyondcompute,powerdelivery,thermalmanagement,connectivity,andsensingmayofferopportunitiesforthemany

analog,microcontroller,andotherfablessplayersstilldefiningwheretheycandifferentiate.Specializationremainsvaluablewhenitsolvesasystem-levelproblemratherthandeliveringabetterlineonadatasheet.

AsenterprisesdecidewheretoinvestinAI,theyneedbetterdemand

signalsthangenericbenchmarksalonecanprovide.

ScalingAlinference4

2.Diverseworkloadscallforspecializedsilicon

MultipleinferencearchitectureswillikelycoexistasdifferentworkloadsplacemateriallydifferentdemandsonAlinfrastructure(Exhibit2).

Exhibit2

Workloadsarediversifying,creatingdemandforspecializedsiliconratherthanvariationsofasinglearchitecture.

Workloadrequirementsandestimatedshareof2030Alinferencetokendemand

Moderaterequirement■HeavierrequirementLargelanguagemodel(LLM)|Non-LLM(13%)Voice

textandcode(82%)

Altraining

Modeltypes

Modelscale

Compute¹(key

characteristic)

Memoryor

characteristic)

networking(key

Servingpattern²

Agentic

LLMs

Multimodalmodels

Diffusionmodels

Wordmodels

Visionmodels

Speechmodels

Recommenda-tionmodels

Reinforcementmodels

10billion-

100billionparameters,upto128,000-tokencontext

window

Low-latencythroughput

Highkey-value-

cachecapacity

lterativeagentloop;tokenbytokenwithcachereuse

Code

High

throughput

Highcapacity

Long-formtokenbytokenwithcachereuse

Chatandsearch

Low-latencythroughput

Lowlatency

Retrieve,theninteractivetokenbytoken

Writing

Under10

billion(short

contextwindow)

High

throughput

High

bandwidth

Tokenbytokenwithcachereuse

Machinetomachine

High-volumerequestandresponse

Media

generation

Moderate

memorycapacity

Long-formtokenbytokenwithcachereuse

PhysicalAl

High

performanceorwattage

Lowlatency

Queuedjobs,oneatatime;3-50steps

Voice

Real-timebidirectionalstreaming

Note:Figuresmaynotsumto100%,becauseofrounding.Optimalarchitecturedependsonmodel,SLO,concurrency,batch

size,andsoftwarestack.

Recognitionandspeechgenerationusespeechmodels

typicallywithoutprefil/decode;easoningtypicallyusesLMs.Requirementsassessedby

dimension;

cross-dimensionaldependenciesnotyetconsidered.

Source:AMD;DeepSeek;Google;JEDEC;LawrenceBerkeley

NationalLaboratory;NVIDIA;OpenA1;UALinkConsortium;UptimeInstitute;McKinsey

analysis

McKinsey&Company

Muchoftoday'sinstalledbasewasdesignedfortrainingandisnowbeingusedforinference.Thatcreatesamismatch,butnottheonespecsheetssuggest.Trainingrewardsrawcompute;decoderewardsmemorybandwidth.Generatingeachtokenmeansstreamingthemodel'sweightsandaccumulatedkey-value(KV)cacheoutofmemorytodocomparativelylittlearithmetic.SobandwidthratherthanpeakFLOPS'setshowfastanodecanserve,andpeakcomputehasbeengrowingroughlytwiceasfastasmemorybandwidthfortwodecades.

1Floatingpointoperationspersecond.

ScalingAlinference5

TheGPU-plus-HBMconfigurationthatdominatesdeploymentstodaywillcontinuetoservealargeshareofworkloads,butitisunlikelytoremainthesingledefaultasinferencerequirementsbecomemorespecific.

Themorelikelyoutcomeisawiderspreadofconfigurations:nodesweightedtowardbandwidthortoward

capacitydependingontheworkload,HBMgenerationsthatwidentheinterfacealongsidecustombasediestunedtoaparticularbuyer'sneeds,advancedpackagingthatshortensthedistancedatatravels,cheaper

memorytiersholdingthecolderpartsofthecache,andSRAM-heavydesignsattackingthebottleneckfromtheoppositeend.Whichbalanceprevailsisunsettledandwilllikelydifferbyworkloadratherthanconverge.

Consideracompliancetoolthatreadsa300contractandreturnsashortlistofflaggedclausesandavoiceassistantthatstreamsaspokenreplytokenbytoken.Thefirstisprefill-heavyandcompute-bound;thesecondisdecode-heavyandbandwidth-bound.Runonthesamehardware,eachpaysforcapabilitiesitdoesnotuse.

Thesamesplitappearsinsideasinglerequest.Prefill,wherethemodelprocessestheprompt,iscompute-intensive.Decode,whereitgeneratestokensoneatatime,dependsmoreonmemorybandwidth.Long

inputsincreaseprefillwork;longoutputsincreasedecodework.Soatextagentandavoiceagent

performingthesametaskmayrequiredifferenthardwarebalances,anddifferentlatencytargetscanfavoraggregatedordisaggregatedserving.

Theindustryisnowdesigningaroundthis.HyperscalersandAllabsarebuildingcustomacceleratorsfor

theirhighest-volumeworkloads.SRAM-basedplayersincludingGroq,Cerebras,SambaNova,andd-Matrixtargetthebandwidth-boundhalfoftheproblem.Andawaveofpairingshasemergedtosplitphasesacrosspurpose-builtsilicon:NVIDIApursuedalicensingagreementandtalenttransferwithGroq,allowingitto

absorbGroq'spurpose-builtinferencetechnology;AWSpairedTrainiumprefillwithCerebrasdecode;AMDpartneredwithCerebras;andIntelandSambaNovaintroducedaninferenceblueprintroutingprefilloGPUsanddecodetoreconfigurabledataflowunits(RDUs).²OpenA'sJalapenotakestheoppositebet:asingle

balancedacceleratorthatactivatestherightmixofcompute,memory,andnetworkperphase,keepingmodelstatelocalratherthanmovingitbetweenhardwarepools.3Inaddition,manyplayersareexploringchiplet-basedarchitecturestoprovidecustomizedsolutionsonastandardcomputeplatform.

There'sonemainriskwithspecialization:Chipcyclesruninyearswhilemodelarchitectureschangein

months,sopurpose-builtsiliconcanarriveoptimizedforworkloadsthathavealreadymoved.Compressingthosecyclesisnowanexplicitindustryobjective.Reinforcement-learningfloorplanninghasbeenin

productionuseacrossseveralgenerationsofGoogle'stensorprocessingunits,andJeffDeanhasarguedpubliclythatAlmethodscouldbringdesigntimelinesfrom18to30monthsdowntothreetosix.4OpenAI'sJalapenoistheclearestproofpointsofar:co-developedwithBroadcom,itmovedfromfirstRTLto

manufacturingtape-outinninemonths-comparedwith18to24monthsforatypicalASICdesignedfromscratch.OpenAluseditsownmodelstoexploreimplementations,compressverificationloops,andoptimizethechip'sarithmeticcircuits.Machine-learning-basedplacementandoptimizationarenowstandard

featuresinthemajorelectronic-design-automationflowsratherthanexperimentaladd-ons,andasetofwell-fundedstart-upsistargetingtherestofthedesignflow.Iftheseeffortsdeliverevenpartofwhattheypromise,teamscoulddesignfortheworkloadstheyactuallyhaveratherthantheonestheypredict,andworkload-specificsiliconwouldbecomefarmoreviable.

2“GroqandNvidiaenternon-exclusiveinferencetechnologylicensingagreementtoaccelerateAlinferenceatglobalscale,Groq,December24,2025;"AWSandCerebrascollaborationaimstosetanewstandardforAlinferencespeedandperfomanceinthecloud,"Cerebras,March13,2026;"AMDandCerebrasannounceindustry-leadingultra-low-latencyandhighthroughputAlinferencesolution,"AMD,July23,2026;

"IntelandSambaNovaadvanceagenticAlwithXeon6,"Intel,April8,2026.

3"Jalapeno'sfirstresultsshowindustry-leadingspeedandefficiencyinAlinference,"OpenAl,August25,2026.

⁴RamishZafar,"FormerGoogleAlbosssaysautomationcouldcutchipdesignteamsfrom150engineerstojustten&developmentfromyearsto3months,"WCCFTECH,September21,2026.

ScalingAlinference6

3.Whereinferencerunsisbasedoneconomics,

latency,anddatacontrol,nottechnology

Computeforinferenceislikelytobeincreasinglydistributedacrosscloud,enterprise,edge,anddevice.

Therightlocationwillvarybyworkload,shapedbydatalocation,economics,latency,reliability,andcontrol.Technicalfeasibilityisrarelyabarrier;it'slargelypossibletoruninferenceineachlocation.Whatdiffersiswhoiswillingtopayforit(Exhibit3).

Exhibit3

Inferencecomputelocationwillbedecidedlessbytechnicalfeasibilityandmorebycommercialclarity.

Whereitruns

Cloud

(eg,frontiermodelserving)

Enterpriseorco-located

(eg,bankscoringfraudon-premises)

Edge(eg,visual

inspectionontheline)

Device(eg,

summarization)

on-phonemessage

Technical

readiness

Commercialclarity

Ready

Settled

Ready

Settled

Ready

Unresolved

Emergingoruse-case

dependent

Partial

McKinsey&Company

Cloud.Cloudwilllikelyremaincentralforfrontiermodels,elasticworkloads,rapidmodelrefreshes,andapplicationswheredataisalreadycentralized.Butitmaynolongerbethedefaultforeveryworkloadaseconomicsandotherrequirementsbecomemoreimportant.

Enterprise.Enterpriseandco-locationinfrastructuremaygrowwheregovernanceandsovereigntyare

critical.Forexample,banksscoringfraudagainstcustomerrecordswilllikelywanttokeepsensitive

customerrecordsontheirpremises.Theseworkloadsstillneeddata-center-classsystems;theysimplysitinprivateclustersratherthanhyperscaleones.

Edge.Edgeservesworkloadsneedingsubstantialcomputenearwheredataisgenerated-suchasa

plantrunningvisualinspectiononeveryunitcomingofftheline,wheremovingthevideocostsmorethanprocessingitlocally.Whileedgeistechnicallyfeasible,noonehassettledwhopays.Telecommunicationsoffersacautionarytale:Operatorsfundedbillionsin4G,consumerspaidamodestmonthlysubscription,andmuchofthevalueaccruedtotheapplicationsbuiltontop.Telcos,utilities,andsiteoperatorswill

hesitatetomovetoedgeuntilthepaymentquestionisclear.

Device.Thisdeploymentoptionwinswhenthemodelfitstheendpoint'smemory,power,andthermalenvelopeandwhenimmediacy,privacy,orofflineoperationmatters.Forexample,aphonesummarizesmessageslocallyandescalatesonlytheharderrequestsupstream.

ScalingAlinference7

4.HardwaredictateswhatAIcomputecandeliver,while

softwaredictateshowmuchcanbecaptured

Acceleratorswillremaincentralgoingforward,buttheymaynolongerdetermineinferenceeconomicsontheirown.Withinsilicon,performanceisincreasinglyshapedbythreeinterdependentelements:Logicperformsthecompute,memoryfeedsit,andnetworkingconnectsandscalesit.Lookingahead,memorybandwidthandnetworkingarelikelytobecomeincreasinglyimportantconstraints-leadingsomeintheindustrytoarguethatthenetworkisbecomingpartofthecomputeritself.

Thatceilingisphysical.AscontextwindowslengthenandagenticworkloadsmultiplyKV-cachetraffic,

memorycapacityandbandwidthsethowmanyconcurrentrequestsanodecanserve.Networkingsets

howfarasinglerequestcanusefullybespreadacrossarackbeforelatencyerodesthegain.Powerdeliveryandthermalmanagement(forexample,liquidcoolingandemergingapproachessuchasmicrofluidics)

increasinglygatedensitybeforecomputedoes.Manyofthelatestsemiconductorinnovations(suchas

siliconphotonicsoradvancedpackaging)attackthesameproblem:theenergyandlatencycostofmovingdataratherthanprocessingit.

Thenear-termcostcurve,though,mightbewoninsoftware.Acodingassistantthatrereadsthesame

repositoryoneveryrequestcanhavemuchofitsprefillcostremovedthroughcachereuse.Acustomer

servicefleetthatroutessimplequeriestoasmallmodelandescalatesonlythemorecomplexoneschangescostperresolutionwithouttouchingthehardware.Batching,scheduling,quantization,anddataplacementbehavethesameway:Gainscompound,andtheymakeexistinghardwarelessexpensivetorun.

Lookingahead,memorybandwidthandnetworkingarelikelytobecomeincreasinglyimportantconstraints—leadingsomeintheindustryto

arguethatthenetworkisbecomingpartofthecomputeritself.

ScalingAlinference8

Theimplicationisthatinferenceinfrastructureisafull-stackoptimizationproblem(Exhibit4).Hardware

defineswhatasystemcandeliver,whilesoftware(andultimatelyenterprisedataandworkflowintegration)determineswhatitactuallydelivers.

Exhibit4

Costperformancerequiresimprovementacrosscompute,memoryandnetworking,andsoftware.

Inferenceoptimizationtechniques

Customsilicon:Design

specializedchipstunedtoinferenceworkloads

Distributedcomputing:

Orchestratemodelexecution(prefillvsdecode)across

differentiatedinfrastructureAdvancednodes(<3

nanometers):Achieveenergyefficiencyanddensity

Advancedpackaging(2.5Dor3DandCoWoS1):Achievehigherbandwidthandchipletscaling

Scale-upinterconnect:Use

high-speedinterconnectsto

minimizeintra-nodecommu-

nicationlatency

Opticalinterconnect

evolution:Deployoptics-

integratedswitches(CPO2)forhigherbandwidthdensityandlowerpowerperbit

CXLandDRAM³tiering:Poolandoffloadkey-value(KV)

datatocheaper,sharedmemorytiers

LPDDR4:Integratelow-powermemoryforenergy-efficientinferencechips

Software

Modeloptimizationtechniques

(eg,quantization,mixtureofexpertsdistillation,KV-cacheoptimizationtocutcomputeandmemorycost)

Softwareschedulingtechniques

(eg,continuousbatching,

topology-awarescheduling)

IChip-on-Wafer-on-Substrate.

Copackagedoptics.

ComputeExpressLinkanddynamicrandom-accessmemory.

4Low-powerdoubledatarate.

McKinsey&Company

5.Specializationonlypaysatscale,makingopenstandardsanecessity

AsAlbecomesembeddedincriticalworkflows,governmentsandenterprisesareplacinggreaterweight

onwheredataisprocessed,whichmodelsareused,whereinfrastructureresides,andwhocanaccesstheoutputs.Foragrowingsetofworkloads,sovereignty,resilience,andregulatoryrequirementshavebecomeasimportantascostandperformanceindeploymentdecisions.

Theeffectisthatrequirementsaredivergingratherthanconverging.Anationalhealthservice,adefenseministry,andaregionalbankmayallwantthesamecapabilitybutunderdifferentconstraintsondata

residency,approvedsuppliers,modelprovenance,andlocalecosystemparticipation.Demandthatonceconvergedonahandfulofhyperscaleregionscouldinsteadresolveintodozensofin-countrydeployments,eachwithitsownspecification.Forsemiconductorcompanies,thatisbothademanddriverandasourceofcomplexity,involvingincrementalbuild-outonfragmentingrequirements.

ScalingAlinference9

FindmorecontentlikethisontheMcKinseyInsightsApp

Scan·Download·Personalize

Sovereigntyisnottheonlyforcethatcouldpullnthisdirection.Ordinarycompetitionwillhaveasimilar

effect:Hyperscalersoptimizingaroundtheirownacceleratorsandsoftware,merchantvendorsdefendingfull-stackplatforms,androboticsandautomotiveplayersdesigningtotheirownlatency,safety,andpowerenvelopesallhavereasonstodifferentiateratherthanconverge.Inbothcases,thelikelyresultisseveral

viableinferencestacks.Sovereigntylayersadditionalconstraintsontopofthedivergencethatcompetitionalreadycreates.

Fragmentationiswhereopenmodelsandstandardsbecomeeconomicallyimportantratherthan

ideological.Open-weightmodelsletanoperatortakethesamecapabilityintoasovereigncloud,aprivatecluster,oranedgedeploymentandtuneit