解决扩展AI内存的能源成本问题

TotalRecall

HowAIIsSuperchargingMemoryDemand

Pleaserefertoimportantdisclosuresonpages44-46.Analystcertificationisonpage44.

WilliamBlairoranaffiliatedoesandseekstodobusinesswithcompaniescoveredinitsresearchreports.Asa

result,investorsshouldbeawarethatthefirmmayhaveaconflictofinterestthatcouldaffecttheobjectivityofthisreport.Thisreportisnotintendedtoprovidepersonalinvestmentadvice.Theopinionsandrecommendationshere-indonottakeintoaccountindividualclientcircumstances,objectives,orneedsandarenotintendedasrecommen-dationsofparticularsecurities,financialinstruments,orstrategiestoparticularclients.Therecipientofthisreportmustmakeitsownindependentdecisionsregardinganysecuritiesorfinancialinstrumentsmentionedherein.

WilliamBlair

AI/INSIGHTS

Contents

Introduction 3

KeyTakeaways 3

BriefHistoryofMemory 5

BitsandBytes:TheMemoryHierarchy 9

RedefiningMemory:TheEmergenceofAI 21

MemoryContentinAIRacks 30

MemoryTotalAddressableMarket 33

Next-GenTechnologies:FurtherExpandingAccesstoMemory 39

Conclusion 42

2

SebastienNaji+12122456508

WilliamBlair

AI/INSIGHTS

Introduction

WhilesomuchoftheAIplatformshifthasbeenfocusedoncomputingdemand,thetraditionallyslowergrowthandlessexcitingmemorymarketisstartingtohaveitsdayinthesun.Advance-mentsinAIcomputeareincreasinglyhinderedbymemorybottlenecks,revitalizingmemoryasanareaofinnovationasvendorsracetodevelophigh-bandwidthsolutionscapableofaddressingthedata-intensiveneedsofAImodels.WithAIinferencedemandstillinaphaseofearlyadoption,webelievethenextseveralyearswillbecharacterizedbyastepwiseincreaseinglobalmemorycapac-itywithnewtechnologiesemergingtodeliverlargevolumesofdatatoAIcomputinginfrastructure.

TheWilliamBlairtechnologyteamhaspreviouslyexploredAI-drivenadvancementsincompute,detailinghowtheAIplatformshiftrequireadeeplyintegratednetworkofnext-generationtech-nologies(FromChipstoSystems:HowAIIsRevolutionizingComputeandInfrastructure)andof-feringaGenAIprimer(GenerativeAI:TheNextFrontierofInnovation).Inthisreport,weofferadetailedexplanationofpast,present,andnext-generationmemorysolutionsandtheevolvingmarketdynamicsasAIfundamentallychangesthispreviouslyquietindustry.

Inconjunctionwiththisreport,weareinitiatingcoverageofthreecompanies:Micron,Rambus,andSiliconMotion.

KeyTakeaways

BandwidththeKeyAIMemoryBottleneck

AIperformanceisincreasinglylimitedbymemorybandwidth,sincelargelanguagemodels(LLMs)requiremovingenormousvolumesofdataforrelativelylittlecomputation..Trainingandinferenceinvolverepeatedlystreamingmassiveamountsofdata—e.g.,modelparameters,context,andKVcache—throughacceleratorswithrelativelylowdatareuse.Asaresult,performanceisoftenlim-itedbyhowfastdatacanbedeliveredratherthanhowmanycomputefloating-pointoperationspersecond(FLOPS)areavailable.Sincethe1990s,processorperformancehasscaledatamuchfasterratethanmemorybandwidth,leadingtothedevelopmentofthe“memorywall.”WhileDRAMband-widthhasimprovedbyafactorof1.6xeverytwoyearsoverthelasttwodecades,computeFLOPshaveincreased3xeverytwoyears,resultinginacompute-memorybandwidthperformancegapthathasreachedamorethan600:1ratio.Asmodelshavegottenlargerovertime,GPUsissuethou-sandsofmemoryrequestsinparallel,quicklysaturatingmemoryinterfaces.Whenmemoryband-widthisinsufficient,expensivecomputeunitssitidleanddrivedowntheutilizationofGPUclusters.

HBMImprovesMemoryAccessforAIAccelerators

Tohelpaddressthememorywallproblem,memoryvendorshavedevelopedanewtypeofdy-namicrandom-accessmemory(DRAM),high-bandwidthmemory(HBM).HBMhelpsaddressthememorywallbyincreasingeffectivememorybandwidth—DRAMmemorystacksareintegratedin-packagewiththeprocessorandconnectedviathousandsofshort,low-powerwiresonasiliconinterposer(TSVs,orthrough-siliconvias).Becauseofitswideinterfaces(16timeswiderthantra-ditionaldoubledatarate[DDR]channels),HBMenablesGPUstoquicklyaccesslargevolumesofdata.WithHBM4,vendorsaredoublingthememorybuscapacityto2,048bits,whichinadditiontoanincreaseinpinspeeds(beyond11GbpsforHBM4)shouldhelpdriveasignificantimprove-mentinmemorybandwidthfornewAIacceleratordeploymentsin2026.

MemoryVendorsShiftingFocusTowardMoreProfitableHBM

ThesuccessofHBM,firstintroducedbySKHynixandfollowedbyofferingsfromMicronandSam-sung,haspushedthemajormemoryvendorstowardHBMbecauseitdelivershigherrevenueandprofitperwafer.Today,asingleHBM3Estack(usedinNvidiaBlackwellGPUs)deliversroughly

SebastienNaji+12122456508

3

WilliamBlair

AI/INSIGHTS

4

SebastienNaji+12122456508

1.2–1.3TB/s,risingtomorethan2.0TB/swithHBM4,versusonlyabout50–100GB/sperDDR5DIMM.Economically,HBMsellsforASPsthatareroughly3-4timesthatoftraditionalDDRDRAM,anddespitehigherTSVandpackagingcostsandloweryields,grossmarginsaremateriallyhigherbecauseAIcustomersvaluebandwidthfarmorethanrawcapacity.WeestimateHBMgrossmar-ginsareinthe55%to65%rangetoday,comparedtohistoricalDRAMgrossmarginsbetween25%and45%.Thisisdrivingadramaticmixshiftincapacityallocationatthemajormemoryvendors.SKHynix,whichcontrolsapproximately60%oftheHBM3market,nowallocatesmorethanhalfofitsDRAMwaferoutputtoHBM,andSamsungisrestructuringitsDRAMroadmaptoprioritizeHBMdespitequalificationchallenges.ThisisresultingintightercapacityformoretraditionalDRAMso-lutions,withcompaniesexitinglower-profitsegmentsentirelytorefocusresources—inDecember2025,MicronannounceditwasterminatingitsCrucialconsumerflashbusiness.

HBMBaseDieBecomingaCriticalValueCaptureBattleground

ThebaselogicdieinHBMhasevolvedfromarelativelysimpleinterfaceintoastrategicallyim-portantcompute-adjacentcomponentwithroomforsignificantcustomization.Thusfar,HBMbasedieshavereliedonoldermemorynodesandhandledbasicfunctionality(e.g.,managingelectricalconnections,translatinghigh-levelGPUinstructions,distributingsignalsacrossTSVs);mostintelligenceresidesintheGPU’smemorycontroller.WithHBM4,thereisashifttoamorecapableandperformantbasediebuiltonleadingprocessornodetechnologies(MicronandSKHynixwillutilizeTSMC).IntheupcominggenerationofHBM,thememorycontrollerwillbeintegratedintothebasedie.ForprocessordesignerslikeNvidia,AMD,andMarvell,thisopensthedoortodeepercodesign,wherecustomizingthebasedieforalargermemorybeachfrontortooffloadfunctionalityfromthecoreprocessorcandrivebetterperformanceforthechipsystem.AsmorevendorsfocusontapingoutcustomHBMbasediesfortheirownprocessors/infrastructurestacks,HBMvendorsmayoverthemediumtolongtermseeeventhispartofthemarketbecomeincreasinglycommoditized.

AIMemoryMomentumExtendstoDRAMWritLarge

WhileHBMcapturesthespotlight,theAIbuildoutisalsopullingthroughsubstantialdemandforhigh-performancenon-HBMDRAM.LargeAIclustersarebuiltonacombinationofmanychipsbe-yondtheGPU,includingCPUs,networkinterfacecards(NICs),dataprocessingunits(DPUs),andstoragecontrollers.Thesesolutionspullthroughsignificantmemorydemand—e.g.,eachNvidiaGB300CPUrequires280GBofDDR5memory(morethan20TBperNVL72rack).Inaddition,newerchipdesignsareincorporatingothertypesofDRAM;Nvidia’snewRubinCPXGPU,whichaddresseslong-contextprompts,swapsusageofmoreexpensiveHBMforlargercapacityGDDR7.Whilebitdemandgrowthshouldbeupdoubledigitsoverthenextfewyears,capacityfortradition-alDRAMistight.WithmemoryvendorsfocusingmoreoftheircapacityonHBM(whichrequires3timesasmanyDRAMwafersforthesamecapacity),ASPsfortraditionalDRAMsolutionshaveseentremendousupwardpressure(TrendForceestimates55%-60%ASPgrowthforconventionalDRAMinthefirstquarterof2026).WeexpecthighpricesforDRAMmemorytoremainthenormforthenextfewyearsasnewmanufacturingcapacitytakesatleastanother18monthstocomeonline.

NANDStoragetoBenefitFromAI

AsinferencebecomesthedominantAIworkload,demandisalsoexpandingtowardNANDandstoragetosupportpersistent,large-scalemodeldata.Inferenceworkloadsrequirefastaccesstomassivenumberofmodelweights,embeddings,retrieval-augmentedgeneration(RAG)datasets,and—critically—persistentorreusableKVcache,allofwhicharequicklyexceedingthecapacityofHBMorsystemDRAM.Thisisdrivinghigherattachratesofhigh-performanceNANDSSDs.Nvid-ia’sInferenceContextMemoryStoragePlatform(announcedinJanuary2026atCES)reinforcestheincreasinglyimportantroleofSSDs,bydecouplingcontextfromHBM/DRAMandenablingstorageasanactivetier.ItusesBlueFieldDPUstoexpandinferencecontexttomemoryform-factorsthathavealowercostperbitthanDRAM,whilemaintainingsystemperformancethrough

WilliamBlair

AI/INSIGHTS

5

SebastienNaji+12122456508

softwareoptimizations.SustainedgrowthforflashstorageshouldbenefitNANDandSSDsfromvendorslikeMicron,SKHynix,Samsung,Kioxia,andSanDisk.Asinferencingisadoptedbyenter-prises,McKinseyestimatesa35%CAGRintheenterpriseSSD(eSSD)marketfrom2024to2030.

MemoryControllersandInterfacesBecomingMoreSophisticated

Memorycontrollersandinterfacetechnologieshaveevolvedfromsimpleinterfacestocritical,high-valuesystemcomponents.Today,memorymanagementtechnologyincreasinglytakesoncompute-adjacentlogicandservesasanactivegatekeeperofAIperformance.Inaddition,memorycontrollers/interfacesmustbeprotocolagnosticandabletomanageheterogenousconnectionstoabroadmixofprocessors(CPUs,GPUs,andDPUs).InDRAM,whilecontrollersremainintegratedintotheprocessor,higherspeedsandnewformfactors(DDR5,MRDIMMs,HBM)requireincreas-inglycomplexPHYsandtiminglogic,drivingupthevalueofmemoryinterfacetechnologies.Mean-while,inNAND,theriseoflessexpensiveanddensersolutions(likequad-levelcell,orQLC)hasforcedcontrollerstocompensateforweakermediacharacteristicsthroughheavierfirmware,sig-nalprocessing,andadaptivealgorithms.Thisgrowingintelligenceshiftsdifferentiationandvaluecapturetowardthecontroller/interfacelayer,allowingvendorslikeRambus,SiliconMotion,Phi-son,andMarvellwithstrongIP,firmware,andsystem-levelexpertisetocommandhighermarginsandembedthemselvesmoredeeplyincustomerplatforms.

Next-GenMemorySolutionsDesignedtoAddressMemoryWall

WhileinnovationslikeHBMhavemadesignificantstridesinscalingmemorybandwidthandca-pacity,theyfallshortofcompletelyfillinginthememorywallgap.Next-generationsolutionshaveemergedtohelpimprovethiscriticalbottleneck.Theseincludecompute-in-memory(allocatingprocessingdirectlytothememorychip),neuromorphicchips(modeledafterhumanneurons,usesspikingneuralnetworkstoprocessinformation),high-bandwidthflash(whichstacksNANDintoanHBM-likestructure),andresistiveRAM(storingdataasresistanceratherthanvoltage).Inaddition,theuseofComputeExpressLink(CXL)isemergingashigher-bandwidthmemoryinterconnectfabricthatallowsCPUsandacceleratorstoshareaccesstomemoryoverPCIe.WehighlightcompanieslikeAnaflash,Astera,BrainChip,Cerebras,Credo,Crossbar,EnchargeAI,Syn-Sense,Syntiant,Upmem,andWeebitNanoashelpingpushtheinnovationenvelopeinnext-genmemorysolutions.

BriefHistoryofMemory

EarlyHistoryofMemoryTechnologies

Sincetheadventofcomputing,memoryhasbeenintegraltotheexecutionofthemostelementaryprograms.Theadvancementofmemoryisprimarilymeasuredthroughtheexpansionofbitsandtheadvancementofreadandwritespeeds.“Bits,”orbinarydigits,arethemostfundamentalunitofdatastorage;thinkofabitastheunitcellofcomputermemory.Bitsareessentiallypatternsoftwopossiblevalues—i.e.,0or1—capableofstoringalldata.Asthisdatagrowincomplexity,thenumberofbitsincreases,thoughthepossiblevalueofthebitisalwayseither0or1.Themorerecognizableunitofstorage—bytes—issimplyasequenceof8bits(e.g.,00000001).Thespeedatwhichthesebitscanbecodedintothememoryisthewritespeed,andthespeedatwhichtheycanberetrievedanddeliveredtotheprocessoristhereadspeed.

Theearliestandmostrudimentaryformofmemorydevelopedinthe1940swasapaperpunchcard.Thecardfunctionedasastandardizedgridstoring960bitsofdata.Whilecostswerelowenoughtokeeppunchcardsrelevantfordecades,thismethodwasinevitablycumbersomeascardswerelimitedincapacityandwrite-once:datacouldnotbere-programmedorrewritten.Giventhatitwouldrequiretensofthousandsofpunchcardstostoredataatthemegabytescale(enoughtostoreasingular,high-qualityiPhonepicture),usedeclinedascomputeneedsadvanced.

6

SebastienNaji+12122456508

WilliamBlairAI/INSIGHTS

Exhibit1

TotalRecall

HistoryofMemorySystems

1965

1964

1998Samsung'sDDR1chip.

1984

1970

1953

2013

2006

IBMSP95SRAMchip.

MOSmemoryinvented.

Intelreleasedthe

Intel1103chip,the

MIT'sWhirlwind

SKHynix

inventedthefirst

becamethefirst

computingsystemto

useJayForrester's

magneticcore

FujioMasuoka

inventedflash

memoryatToshiba.

AWSlaunched

firstcommercially

availableDRAMIC

chip.

generationof

high-bandwidth

memory(HBM).

cloud-based

services,redefining

memoryneeds.

memory.

2010-Present

1940s1950s1960s1970s

2000s

1980s1990s

2000USBflashdrives.

2020OpenAI'sGPT-3.

024HBMEreleased.

J.PresperEckert&

JohnMauchly

inventeddelay-line

memory,later

integratedintoearly

computers,EDVAC

andUNIVACI.

IBMlaunchedthe

System370Model

IBM’sRAMAC

computer

harnesses

emerginghard

diskdrives.

Samsung

145computer,the

introduced

firstIBMcomputerto

synchronous

useall

DRAM,introducing

semiconductor

thenextgeneration

memory.

ofmainmemory.

Sources:WilliamBlairEquityResearch,ComputerHistoryMuseum(CHM),Intel,IBM

WilliamBlair

AI/INSIGHTS

7

SebastienNaji+12122456508

In1944,J.PresperEckert’sdevelopmentofdelay-linememorybecamethefirstwidelyacceptedmemorysystem,notablyusedinearlycomputerssuchasEDVAC.Delay-linememoryreliedonpressurewavepropagationsforreadingandwritingthedata.Thereweretwokeydetrimentstothissystem:1)thesequentialnature(orcyclic-accessmemory)limitedCPUaccessto1bitatatime,and2)storagedensitywasrestrictedbythephysicallimitofwavepropagations.Delay-linememorywasrenderedobsoleteinthe1950s,illustratingtwoimportantpillarsofsubsequentmemorytechnologydevelopment:movementtowardrandom-accessmemory(RAM)anddevel-opingascalable,high-capacitymethodofbitstorage.

MagnetismandStrivingforRAM

Aptlynamed,magneticcorememoryusedmagneticdonuts(orcores)connectedbywireinter-connects;readingandwritingwerecompletedusingelectricalcurrentsinplaceofdelay-linememory’swavepropagations.Keyadvancementsofmagneticcorememorydirectlysolvedtheshortfallsofdelay-linememory,i.e.,randomaccesstoanybit(introducingRAM)andscaledcapac-ityachievedbystackingthegrids.Magneticcorememorywasfirstappliedin1953toMIT’sWhirl-windcomputer,capableofstoringroughly4KBofmemory,aboutone-millionthofthememoryofamodernPC.Magneticcorememoryremainedtheindustrystandardfortwodecades,cuttingthecostperbitto100thoftheinitialcostandincitingcontemporarymemoryaccessmethods,RAM.

MagnetismwasseparatelyyetsimilarlyleveragedbyEckertin1951toinscribethebinarycodeintotheproprietarymemorystoragesystemhedevelopedforUNIVAC,magnetictapememory.Thecheapandcompactnatureofthemagnetictapepermitteddataarchiving,butaccessrequiredtherotationofhundredsoffeetoftape.Thisso-calledseekissuewastemporarilyamelioratedwiththedevelopmentofthemagneticdrum,whichusedarevolvingbarreltoreduceseektimes.Thoughonlyexperiencingashortstintofcommercialization,themagneticbarreldirectlyledtoitssuccessor,theharddisk.

EvolutionofStorage:AdvancingtoSolidStateSolutions

Developedin1956,theharddisk’sconfigurationofacentralaxiswithrotatingplattersmirroredthatofthemagneticdrum.Bynatureofthediskstructure,thelargesurfaceareaallowedstackingtoexponentiallyexpandthecapacity.Readandwriteheadstravelingalongsidethelengthofthestackcouldwedgebetweenthespinningdisks,reducingtheseektimetofindtheindicatedbit.Whilespeedwaslimitedtotherotationtimeonthedisk,thismechanismwasrelativelyfastforstorage,enablingitsdominancebythe1970s.Thefloppydiskoperatedinmuchthesamemanner,butonadifferentmagneticmedium.Bythe1990s,thefloppydiskwasthepreferredmechanismofportablestorage.

Tobrieflydivergefrommagneticinscription,opticalstoragedevices(mostnotablyCDsandDVD),becameubiquitousformusic-andentertainment-specificstorageinthe1990s.Bitswerestoredasdivotsthatreflectedlightsignalscapturedbytheopticaldetector.Thecurrentlandscapeofmem-orystorageisdefinedbysolidstatetechnologies.Byreplacingthemovingpartsinthedevicewithintegratedcircuits(ICs),seektimebecameessentiallynegligible,underscoringtheprominenceandsubsequentriseofsemiconductormemory.

RiseandReignofIntegratedCircuits

Semiconductormemoryenteredthescenein1964withFairchildSemiconductor’s(nowOnsemi)developmentofmetal-oxide-semiconductorfield-effecttransistors(MOSFETs),atechnologicaladvancementthatpermanentlytiltedthescalestowardsemiconductormemory.SemiconductormemorystoresbitsofdatawithinMOSmemorycellscomposedofoneormoretransistorsonasiliconchip.EachbitofdataislocatedataspecifiedaddressthatisquicklyrecognizableandretrievablebytheCPU.Theongoingdominanceofsemiconductormemorycommencedintheearly1970s.

WilliamBlair

AI/INSIGHTS

Dependingontheconfigurationofthememorycell,RAMcaneitherbestatic(SRAM)ordynamic(DRAM).CommercialuseofSRAMcommencedin1965withIBM’sintroductionoftheSP95SRAMchip.DevelopmentofDRAMmemorycellsfollowedwiththe1970debutofthefirstcommercialDRAMICchip,theIntel1103.DRAMdevelopmentwasfurtherspecializedwiththeintroductionofsynchronousdynamicRAM(SDRAM)bySamsungin1992.

In1998,DRAMtechnologieswererevolutionizedwiththeintroductionofSamsung’sMbitDDRSDRAM.Intuitively,DDRSDRAMsarecapableofdoublethebandwidth,quicklybecomingthepreferredandmostusedformofvolatilemainmemory.SuccessivegenerationsofDDRSDRAM—i.e.,DDR2(2003),DDR3(2007),DDR4(2014),andDDR5(2020)—haverepresentedasubstan-tialbandwidthimprovement(seeexhibit2).DDR5boastsmemorycapacityintherangeof64to256+GB,roughlyamillion-timescapacityimprovementfromdelay-linememorywithkilobitsofcapacity.Themostcutting-edgemainmemorysolution,optimizedforAI-specificuse-cases,isHBM,introducedbySKHynixin2013,thenimplementedinAMDFijiGPUstwoyearslater.

Exhibit2

TotalRecall

DramaticGenerationalGainsAcrossDRAMandNAND

300

Capacity/die(GB)Capacity/DIMM(GB)

250

200

150

100

50

0

300

250

200

150

100

50

0

DDR5XXXXXX

——DDRSDRAM,GB/DIMM

DDR4

DDR1DDR2DDR3

218-layer,3DNAND

——NAND,

capacity/die

Planar,500-50

nm

TLCNAND15nm,2DNANDMLCNAND

Source:WilliamBlairEquityResearch,TechInsights

WhilemainmemoryisalmostentirelyvolatileRAM,itisessentialtoacknowledgetheconcurrentdevelopmentofnonvolatile,read-onlymemory(ROM).Themodernbasisofnonvolatilememorywasestablishedwiththedevelopmentoffloatinggatememorycellsin1967byDawonKahngandSimonSzeatBellLabs(formerlyAlcatel-Lucant,nowNokiaBellLabs).Thisfloatinggatetechnol-ogypermittedToshiba’sdevelopmentofNORandNANDflashin1984and1987.

8

SebastienNaji+12122456508

WilliamBlair

AI/INSIGHTS

9

SebastienNaji+12122456508

NORandNAND,namedafterthelogicgatestheyuse(NOTORandNOTAND,respectively),differintheirmemorycellconnectionsandthustheirreadingandwritingcapabilities.NAND,themostpopularnonvolatileflashmemory,issequentialaccessbynatureoftheserialconnectionofthememorycells,sacrificingreadingspeedfordatadensity.NOR,ontheotherhand,canaccessdatainparallelbuthaslowerstoragedensitythanNAND.

In2013,Samsungwasthefirsttocommercialize3DNANDflashwithits24-layerV-NANDtechnol-ogy.Insteadofcontinuingtoshrinkplanar(2D)NANDtransistors—whichwasbecomingphysicallydifficult—Samsungstackedmemorycellsvertically,massivelyimprovingdensityandenduranceandallowingcontinuedgrowthinflashcapacity.

BitsandBytes:TheMemoryHierarchy

Thememoryhierarchy(exhibit3)isthewaycomputingsystemsorganizememoryintolayersthattradeoffspeed,capacity,cost,andproximitytotheprocessortodeliverhighperformanceef-ficiently.Atthetopareregistersandcaches(L1–L3),whichareextremelyfastbuttinyandexpen-sive,sittingdirectlyonorneartheCPU/GPU.Nextismainmemory(DDRorHBM),whichismuchlargerandlessexpensiveperbitbutslowerandfartheraway.Atthebottomarestoragelayers(solid-statedrive[SSD],thenharddiskdrive[HDD]orobjectstorage),whichoffermassivecapac-ityataverylowcostbutwithorders-of-magnitudehigherlatency.

Exhibit3

TotalRecall

CPUCache

HBM

MainMemory

SolidStateMemory

MagneticMemory

TheMemoryHierarchy

CP

U

•

•

Registers

Mostexpensive($1-$10/bit),smallcapacity(bytestoKBs),fastest(<1ns)

•

•

SRAM,L1-3caches

Veryexpensive($0.10-$1/bit),lowcapacity(KBs-MBs),superfast(1-20ns)

•HBM3elatest-generation(8-12stackedDRAMdies)

•Costly,highercapacity(24-36GB/stack),ultrahighbandwidth(1.2TB/s)

•DRAM(SD-RAM,DDR-SDRAM)

•Moderatelypriced($0.0005–$0.01/bit),averagecapacity(4-128GB),fast(~50–100ns)

•SSD/NANDFlash,nonvolatilestorage

•Cheap($0.00005–$0.001/bit),highcapacity(100GB-10TB),avgspeed(~50–100µs)

•Mechanicalharddrives

•Cheapest($0.00001–$0.0001/bit),highestcapacity(100GB-10TB),slow(~5–10ms)

Sources:WilliamBlairEquityResearch,EngineeringLibreTexts

Themostfundamentaldistinctioninthememoryhierarchyisbetweeninternalandexternalmem-ory.Asthenamessuggest,internalmemorystoresdatainsidetheserver/computerandexternalmemoryreliesondatastoredinanauxiliarylocation.

Notethat,whilerelated,theseparationbetweeninternalandexternalmemoryisnotsynonymouswithvolatileversusnonvolatilesystems.Volatilememory—themostubiquitousformofmainmemory—requirespowertomaintainthestoreddata.Incontrast,nonvolatilememorycanretaininformationpermanentlyintheabsenceofpower.Volatilememorytendstobeusedforinternal

WilliamBlair

AI/INSIGHTS

10

SebastienNaji+12122456508

memory(DRAM),whilenonvolatilememoryistypicallyusedforSSDsandstorage.Thatsaid,non-volatileinternalmemorydoesexistintheformofROMandembeddedflashmemorysystems.Likewise,volatileexternalmemoryalsoexists,thoughthesesystemsarethinlyused.

Exhibit4

TotalRecall

SystemMemoryFlowchart

SystemMemory

InternalMemory

SolidStateDrives

(SSDs)

3DNANDflashmodules

HardDiskDrives

(HDDs)

RandomAccessMemory(RAM)

ExternalMemory

Read-OnlyMemory

(ROM)

HAMRdrives;highlatency/capacity

DynamicRAM

(DRAM)

StaticRAM(SRAM)

5nmSRAM,

cacheuse-case

HBM

DDRSDRAM

HBM4;AIworkhorse

DDR5;ubiquitousmainmemory

Source:WilliamBlairEquityResearch,PrincetonDepartmentofComputerScience

InternalMemory

Internalmemoryisbuiltintothecomputer/serverandusedforeverydayoperations.Mainmem-oryiscomposedofbytes.Eachbyterepresentsasequenceofeightbits,themostfundamentalmemoryunit.Abitcanbeoneofonlytwovalues,eitherazeroorone(determinedbythevoltageofeachmemorycell).Eachbyte(i.e.,00000001or0100010)representsadistinctaddressthatisretrievablebytheCPUduringthe“fetch”stageofrunningaprogram.

Traditionally,whenaCPUneedsdata,itissuesaloadrequestthatisfirstcheckedagainstitson-chipcaches(L1,thenL2andL3).Ifthedataisnotfoundthere,therequestisforwardedbytheintegratedmemorycontrollertomainmemoryatthespecifiedaddress.Theretrieveddataarereturnedoverthememoryinterconnecttotheprocessor,placedintothecachehierarchy,andthenconsumedbytheexecutionunits.Cachesaresmall,fastmemorystructuresthatexploitdatalocal-ity,enablingtheCPUtoavoidrepeatedlyaccessingslowermainmemory.

WilliamBlair

AI/INSIGHTS

11

SebastienNaji+12122456508

Exhibit5

TotalRecall

DataRetrievalProcess

Register

Cache(L1-L3)

Bitflowthroughmemorybus

CPU

RAM

(MainMemory)

Storage

(SecondaryMemory)

Sources:WilliamBlairEquityResearch,JournalofInformationSystemsEngineering&Management

Registers

Registersarethesmallestandfastesttypeofmemorysystems,withmuchlowerlatencycomingattheexpenseofamuchhigherpriceperbit.TheregistersarelocatedinsidetheCPUcoresthem-selves,witheachcoreassignedasetofregisters.Toquantifythe“superfast”speedbetweentheCPUandregisters,wecanuseaCPUclockcycle,whichisthetimeittakestoperformoneoperation.Theregister’slatencycorrespondstolessthan1CPUcycleorroughly0.3to1nanosecondoflatency.

Fora64-bitx86CPUarchitecturethereare16general-purposeregisters,eachcapableofstoring64bits(or8bytes),totaling128bytesperCPU.Tocontextualize,128