49-The Stanford DASH Multiprocessor.pdf

TheStanfordDashMultiprocessorDanielLenoski,JamesLaudon,KouroshGharachorloo,WolfDietrichWeber,AnoopGupta,JohnHennessy,MarkHorowitz,andMonicaS.LamStanfordUniversityDirectorybasedcachecoherencegivesDashtheeaseofuseofsharedmemoryarchitectureswhilemaintainingthescalabilityofmessagepassingmachines.heComputerSystemsLaboratoryatStanfordUniversityisdevelopingasharedmemorymultiprocessorcalledDashanabbreviationforDirectoryArchitectureforSharedMemory.Thefundamentalpremisebehindthearchitectureisthatitispossibletobuildascalablehighperformancemachinewithasingleaddressspaceandcoherentcaches.TheDasharchitectureisscalableinthatitachieveslinearornearlinearperformancegrowthasthenumberofprocessorsincreasesfromafewtoafewthousand.Thisperformanceresultsfromdistributingthememoryamongprocessingnodesandusinganetworkwithscalablebandwidthtoconnectthenodes.Thearchitectureallowsshareddatatobecached,therebysignificantlyreducingthelatencyofmemoryaccessesandyieldinghigherprocessorutilizationandhigheroverallperformance.Adistributeddirectorybasedprotocolprovidescachecoherencewithoutcompromisingscalability.TheDashprototypesystemisthefirstoperationalmachinetoincludeascalablecachecoherencemechanism.Theprototypeincorporatesupto64highperformanceRISCmicroprocessorstoyieldperformanceupto1.6billioninstructionspersecondand600millionscalarfloatingpointoperationspersecond.Thedesignoftheprototypehasprovideddeeperinsightintothearchitecturalandimplementationchallengesthatariseinalargescalemachinewithasingleaddressspace.Theprototypewillalsoserveasaplatformforstudyingrealapplicationsandsoftwareonalargeparallelsystem.ThisarticlebeginsbydescribingtheoverallgoalsforDash,themajorfeaturesofthearchitecture,andthemethodsforachievingscalability.Next,wedescribethedirectorybasedcoherenceprotocolindetail.Wethenprovideanoverviewoftheprototypemachineandthecorrespondingsoftwaresupport,followedbysomeMarch1992preliminaryperformancenumbers.ThearticleconcludeswithadiscussionofrelatedworkandthecurrentstatusoftheDashhardwareandsoftware.TheDashteamManygraduatestudentsandfacultvmemberscontributedtotheDashproject.ThePhDstudentsareDanielLenoskiandJamesLauDashprojectOverviewTheoverallgoaloftheDashprojectistoinvestigatehighlyparallelarchitectures.Forthesearchitecturestoachievewidespreaduse,theymustrunavarietyofapplicationsefficientlywithoutimposingexcessiveprogrammingdifficulty.Toachievebothhighperformanceandwideapplicability,webelieveaparallelarchitecturemustprovidescalabilitytosupporthundredstothousandsofprocessors.highperformanceindividualprocessors,andasinglesharedaddressspace.Thegapbetweenthecomputingpowerofmicroprocessorsandthatofthelargestsupercomputersisshrinking.whilethepriceiperformanceadvantageofmicroprocessorsisincreasing.Thisclearlypointstousingmicroprocessorsasthecomputeenginesinamultiprocessor.Thechallengeliesinbuildingamachinethatcanscaleupitsperformancewhilemaintainingtheinitialprice/performanceadvantageoftheindividualprocessors.Scalabilityallowsaparallelarchitecturetoleveragecommoditymicroprocessorsandsmallscalemultiprocessorstobuildlargerscalemachines.Theselargermachinesoffersubstantiallyhigherperformance,whichprovidestheimpetusforprogrammerstoporttheirsequentialapplicationstoparallelarchitecturesinsteadofwaitingforthenexthigherperformanceuniprocessor.Highperformanceprocessorsareimportanttoachievebothhightotalsystemperformanceandgeneralapplicability.Usingthefastestmicroprocessorsreducestheimpactoflimitedorunevenparallelisminherentinsomeapplications.Italsoallowsawidersetofapplicationstoexhibitacceptableperformancewithlesseffortfromtheprogrammer.Asingleaddressspaceenhancestheprogrammabilityofaparallelmachinebyreducingtheproblemsofdatapartitioninganddynamicloaddistribution,twoofthetoughestproblemsinprogrammingparallelmachines.Thesharedaddressspacealsoimprovessupportforautomaticallyparallelizingcompilers,standardoperatingsystems,multiprodonDasharchitectureandhardwaredesignKouroshGharachorlooDasharchitectureandconsistencymodelsWolfDietrichWeberDashsimulatorandscalabledirectoriesTrumanJoeDashhardwareandprotocolverificationtoolsLuisStevensoperatingsystemHelenDavisandStephenGoldschmidttracegenerationtools,synchronizationpatterns,localitystudiesToddMowryevaluationofprefetchoperationsAaronGoldbergandMargaretMartonosiperformancedebuggingtoolsTomChanakmeshroutingchipdesignRichardSimonisyntheticloadgeneratoranddirectorystudiesJosepTorrellassharingpatternsinapplicationsEdwardRothberg,JaswinderPalSingh,andLarrySouleapplicationsandalgorithmdevelopment.StaffresearchengineerDavidNakahiracontributedtothehardwaredesign.ThefacultyassociatedwiththeprojectareAnoopGupta,JohnHennessy,MarkHorowitz,andMonicaLam.gramming,andincrementaltuningofparallelapplicationsfeaturesthatmakeasingleaddressspacemachinemucheasiertousethanamessagepassingmachine.Cachingofmemory,includingsharedwritabledata,allowsmultiprocessorswithasingleaddressspacetoachievehighperformancethroughreducedmemorylatency.Unfortunately,cachingshareddataintroducestheproblemofcachecoherenceseethesidebarandaccompanyingfigure.Whilehardwaresupportforcachecoherencehasitscosts,italsooffersmanybenefits.Withouthardwaresupport,theresponsibilityforcoherencefallstotheuserorthecompiler.Exposingtheissueofcoherencetotheuserwouldleadtoacomplexprogrammingmodel,whereusersmightwellavoidcachingtoeasetheprogrammingburden.Handlingthecoherenceprobleminthecompilerisattractive.butcurrentlycannotbedoneinawaythatiscompetitivewithhardware.Withhardwaresupportedcachecoherence,thecompilercanaggressivelyoptimizeprogramstoreducelatencywithouthavingtorelypurelyonaconservativestaticdependenceanalysis.Themajorproblemwithexistingcachecoherentsharedaddressmachinesisthattheyhavenotdemonstratedtheabilitytoscaleeffectivelybeyondafewhighperformanceprocessors.Todate,onlymessagepassingmachineshaveshownthisability.Webelievethatusingadirectorybasedcoherencemechanismwillpermitsingleaddressspacemachinestoscaleaswellasmessagepassingmachines,whileprovidingamoreflexibleandgeneralprogrammingmodel.DashsystemorganizationMostexistingmultiprocessorswithcachecoherencerelyonsnoopingtomaintaincoherence.Unfortunately,snoopingschemesdistributetheinformationaboutwhichprocessorsarecachingwhichdataitemsamongthecaches.Thus,straightforwardsnoopingschemesrequirethatallcachesseeeverymemoryrequestfromeveryprocessor.Thisinherentlylimitsthescalabilityofthesemachinesbecausethecommonbusandtheindividualprocessorcacheseventuallysaturate.WithtodayshighperformanceRISCprocessorsthissaturationcanoccurwithjustafewprocessors.Directorystructuresavoidthescalabilityproblemsofsnoopyschemesbyremovingtheneedtobroadcasteverymemoryrequesttoallprocessorcaches.Thedirectorymaintainspointerstotheprocessorcachesholdingacopyofeachmemoryblock.Onlythecacheswithcopiescanbeaffectedbyanaccesstothememoryblock,andonlythosecachesneedbenotifiedoftheaccess.Thus,theprocessorcachesandinterconnectwillnotsaturateduetocoherencerequests.Furthermore.directorybasedcoherenceisnotdependentonanyspecificinterconnectionnetworklikethebususedbymostsnoopingschemes.Thesamescalable,lowlatencynetworkssuchasOmeganetworksorknaryncubesusedbynoncachecoherentand64COMPUTERCachecoherenceCachecoherenceproblemscanariseinsharedmemorymultiprocessorswhenmorethanoneprocessorcacheholdsacopyofadataitema.Uponawrite,thesecopiesmustbeupdatedorinvalidatedb.Mostsystemsuseinvalidationsincethisallowsthewritingprocessortogainexclusiveaccesstothecachelineandcompletefurtherwritesintothecachelinewithoutgeneratingexternaltrafficc.Thisfufthercomplicatescoherencesincethisdirtycachemustrespondinsteadofmemoryonsubsequentaccessesbyotherprocessorsd.Smallscalemultiprocessorsfrequentlyuseasnoopycachecoherenceprotocol,whichreliesonallcachesmonitoringthecommonbusthatconnectstheprocessorstomemory.Thismonitoringallowscachestoindependentlydeterminewhentoinvalidatecachelinesb,andwhentointervenebecausetheycontainthemostuptodatecopyofagivenlocationd.Snoopyschemesdonotscaletoalargenumberofprocessorsbecausethecommonbusorindividualprocessorcacheseventuallysaturate,sincetheymustprocesseverymemoryrequestfromeveryprocessor.onmemoryrequestsbykeepingtrackofwhichcachesholdeachmemoryblock.AsimpledirectorystructurefirstproposedbyCensierandFeautrierhasonedirectoryentryperblockofmemorye.Eachentrycontainsonepresencebitperprocessorcache.Inaddition,astatebitindicateswhethertheblockisuncached,sharedinmultiplecaches,orheldexclusivelybyonecachethatIS,whethertheblockisdirty.Usingthestateandpresencebits,thememorycantellwhichcachesneedtobeinvalidatedwhenalocationiswrittenb.Likewise,thedirectoryindicateswhethermemoryscopyoftheblockisuptodateorwhichcacheholdsthemostrecentcopyd.Ifthememoryanddirectoryarepartitionedintoindependentunitsandconnectedtotheprocessorsbyascalableinterconnect,thememorysystemcanprovidescalablememorybandwidth.ThedirectoryrelievestheprocessorcachesfromsnoopingReferences1.J.ArchibaldandJ.L.Baer,CacheCoherenceProtocolsEvaluationUsingaMultiprocessorSimulationModel,ACMTrans.ComputerSystems,Vol.4,No.4,Nov.1986,pp.273298.2.L.CensierandP.Feautrier,ANewSolutiontoCoherenceProblemsinMulticacheSystems,/E€€Trans.Computers,Vol.C27,No.12,Dec.1978,pp.1,1121,118.Store3,ACacheCache4LoadAerdDataStatebitPresencebitsemessagepassingmachinescanbeemployed.Theconceptofdirectorybasedcachecoherenceisnotnew.Itwasfirstproposedinthelate1970s.However,theoriginaldirectorystructureswerenotscalablebecausetheyusedacentralizeddirectorythatquicklybecameabottleneck.TheDasharchitectureovercomesthislimitationbypartitioninganddistributingthedirectoryandmainmemory,andbyusinganewcoherenceprotocolthatcansuitablyexploitdistributeddirectories.Inaddition,DashprovidesseveralothermechanismstoMarch199265reduceandhidethelatencyofmemoryoperations.Figure1showsDashshighlevelorganization.Thearchitectureconsistsofanumberofprocessingnodesconnectedthroughdirectorycontrollerstoalowlatencyinterconnectionnetwork.Eachprocessingnode,orcluster,consistsofasmallnumberofhighperformanceprocessorsandaportionofthesharedmemoryinterconnectedbyabus.Multiprocessingwithintheclustercanbeviewedeitherasincreasingthepowerofeachprocessingnodeorasreducingthecostofthedirectoryandnetworkinterfacebyamortizingitoveralargernumberofprocessors.Distributingmemorywiththeprocessorsisessentialbecauseitallowsthesystemtoexploitlocality.Allprivatedataandcodereferences,alongwithsomeofthesharedreferences,canbemadeloo00Figure1.TheDasharchitectureconsistsofasetofclustersconnectedbyageneralinterconnectionnetwork.Directorymemorycontainspointerstotheclusterscurrentlycachingeachmemoryline.ScalabilityoftheDashapproachWehaveoutlinedwhywebelieveasingleaddressspacemachinewithcachecoherenceholdsthemostpromisefordeliveringscalableperformancetoawiderangeofapplications.Here,weaddressthemoredetailedissuesinscalingsuchadirectorybasedsystem.Thethreeprimaryissuesareensuringthatthesystemprovidesscalablememorybandwidth,thatthecostsscalereasonably,andthatmechanismsareprovidedtodealwithlargememorylatencies.Scalabilityinamultiprocessorrequiresthetotalmemorybandwidthtoscalelinearlywiththenumberofprocessors.Dashprovidesscalablebandwidthtodatacaltothecluster.Thesereferencesavoidarchitectureissimilartomanyscalableobjectsresidinginlocalmemorybydisthelongerlatencyofremotereferencesmessagepassingmachines.Whilenottributingthephysicalmemoryamongandreducethebandwidthdemandsonoptimizedtodoso,Dashcouldemulatetheclusters.Fordataaccessesthatmusttheglobalinterconnect.Exceptforthesuchmachineswithreasonableeffibeservicedremotely,Dashusesascaldirectorymemory,theresultingsystemciency.ableinterconnectionnetwork.Support100goAverageinvalidationspersharedwrite0.71Averageinvalidationspersharedwrite0.39looM8079270E26055084053016.E60260vv550cLmc.6.3.1.1.1.1.O.O.1.3.2.1.1.1.o.o.401234567891010012345678910210InvalidationsInvalidationsFigure2.CacheinvalidationpatternsforMP3DaandPThorb.MP3Dusesaparticlebasedsimulationtechniquetodeterminethestructureofshockwavescausedbyobjectsflyingathighspeedintheupperatmosphere.PThorisaparallellogicsimulatorbasedontheChandyMisraalgorithm.66COMPUTERofcoherentcachescouldpotentiallycompromisethescalabilityofthenetworkbyrequiringfrequentbroadcastmessages.Theuseofdirectories,however,removestheneedforsuchbroadcastsandthecoherencetrafficconsistsonlyofpointtopointmessagestoclustersthatarecachingthatlocation.Sincetheseclustersmusthaveoriginallyfetchedthedata,thecoherencetrafficwillbewithinsomesmallconstantfactoroftheoriginaldatatraffic.Infact,sinceeachcachedblockisusuallyreferencedseveraltimesbeforebeinginvalidated,cachingnormallyreducesoverallglobaltrafficsignificantly.Thisdiscussionofscalabilityassumesthattheaccessesareuniformlydistributedacrossthemachine.Unfortunately,theuniformaccessassumptiondoesnotalwaysholdforhighlycontendedsynchronizationobjectsandforheavilyshareddataobjects.Theresultinghotspotsconcentratedaccessestodatafromthememoryofasingleclusteroverashortdurationoftimecansignificantlyreducethememoryandnetworkthroughput.Thereductionoccursbecausethedistributionofresourcesisnotexploitedasitisunderuniformaccesspatterns.Toaddresshotspots,Dashreliesonacombinationofhardwareandsoftwaretechniques.Forexample,Dashprovidesspecialextensionstothedirectorybasedprotocoltohandlesynchronizationreferencessuchasqueuebasedlocksdiscussedfurtherinthesection,Supportforsynchronization.Furthermore,sinceDashallowscachingofsharedwritabledata,itavoidsmanyofthedatahotspotsthatoccurinotherparallelmachinesthatdonotpermitsuchcaching.Forhotspotsthatcannotbemitigatedbycaching,somecanberemovedbythecoherenceprotocolextensionsdiscussedinthesection,Updateanddeliveroperations,whileotherscanonlyberemovedbyrestructuringatthesoftwarelevel.Forexample,whenusingaprimitivesuchasabarrier,itispossibleforsoftwaretoavoidhotspotsbygatheringandreleasingprocessorsthroughatreeofmemorylocations.Regardingsystemcosts,amajorscalabilityconcernuniquetoDashlikemachinesistheamountofdirectorymemoryrequired.Ifthephysicalmemoryinthemachinegrowsproportionallywiththenumberofprocessingnodes,thenusingabitvectortokeeptrackofallclusterscachingamemoryblockdoesnotscalewell.ThetotalamountofdirectorymemoryneededisPxMILmegabits,wherePisthenumberofclusters,Misthemegabitsofmemorypercluster,andI,isthecachelinesizeinbits.Thus,thefractionofmemorydevotedtokeepingdirectoryinformationgrowsasPIL.Dependingonthemachinesize,thisgrowthmayormaynotbetolerable.Forexample,consideramachinethatcontainsupto32clustersofeightprocessorseachandhasacachememorylinesizeof32bytes.Forthismachine,theoverheadfordirectorymemoryisonly12.5percentofphysicalmemoryasthesystemscalesfromeightto256processors.Thisiscomparablewiththeoverheadofsupportinganerrorcorrectingcodeonmemory.Forlargermachines.wheretheoverheadwouldbecomeintolerable,severalalternativesexist.First,wecantakeadvantageofthefactthatatanygiventimeamemoryblockisusuallycachedbyaverysmallnumberofprocessors.Forexample,Figure2showsthenumberofinvalidationsgeneratedbytwoapplicationsrunonasimulated32processormachine.Thesegraphsshowthatmostwritescauseinvalidationstoonlyafewcaches.Wehaveobtainedsimilarresultsforalargenumberofapplications.Consequently,itispossibletoreplacethecompletedirectorybitvectorbyasmallnumberofpointersandtousealimitedbroadcastofinvalidationsintheunusualcasewhenthenumberofpointersistoosmall.Second,wecantakeadvantageofthefactthatmostmainmemoryblockswillnotbepresentinanyprocessorscache,andthusthereisnoneedtoprovideadedicateddirectoryentryforeverymemoryblock.Studieshaveshownthatasmalldirectorycacheperformsalmostaswellasafulldirectory.Thesetwotechniquescanbecombinedtosupportmachineswiththousandsofprocessorswithoutundueoverheadfromdirectorymemory.Theissueofmemoryaccesslatencyalsobecomesmoreprominentasanarchitectureisscaledtoalargernumberofnodes.Therearetwocomplementaryapproachesformanaginglatencymethodsthatreducelatencyandmechanismsthathelptolerateit.Dashusesbothapproaches,thoughourmainfocushasbeentoreducelatencyasmuchaspossible.Althoughlatencytoleratingtechniquesareimportant.theyoftenrequireadditionalapplicationparallelismtobeeffective.HardwarecoherentcachesprovidetheprimarylatencyreductionmechanisminDash.Cachingshareddatasignificantlyreducestheaveragelatencyforremoteaccessesbecauseofthespatialandtemporallocalityofmemoryaccesses.Forreferencesnotsatisfiedbythecache,thecoherenceprotocolattemptstominimizelatency,asshowninthenextsection.Furthermore,aspreviouslymentioned,wecanreducelatencybyallocatingdatatomemoryclosetotheprocessorsthatuseit.Whileaveragememorylatencyisreduced,referencesthatcorrespondtointerprocessorcommunicationcannotavoidtheinherentlatenciesofalargemachine.InDash,thelatencyfortheseaccessesisaddressedbyavarietyoflatencyhidingmechanisms.Thesemechanismsrangefromsupportofarelaxedmemoryconsistencymodeltosupportofnonblockingprefetchoperations.TheseoperationsaredetailedinthesectionsonMemoryconsistencyandPrefetchoperations.Wealsoexpectsoftwaretoplayacriticalroleinachievinggoodperformanceonahighlyparallelmachine.Obviously,applicationsneedtoexhibitgoodparallelismtoexploittherichcomputationalresourcesofalargemachine.Inaddition,applications,compilers,andoperatingsystemsneedtoexploitcacheandmemorylocalitytogetherwithlatencyhidingtechniquestoachievehighprocessorutilization.Applicationsstillbenefitfromthesingleaddressspace,however,becauseonlyperformancecriticalcodeneedstobetunedtothesystem.Othercodecanassumeasimpleuniformmemorymodel.TheDashcachecoherenceprotocolWithintheDashsystemorganization,thereisstillagreatdealoffreedominselectingthespecificcachecoherenceprotocol.ThissectionexplainsthebasiccoherenceprotocolthatDashusesfornormalreadandwriteoperations,thenoutlinestheresultingmemoryconsistencymodelvisibletotheprogrammerandcompiler.Finally,itdetailsextensionstotheprotocolthatsupportlatencyhidingandefficientsynchronization.March199261Memoryhierarchy.Dashimplementsaninvalidationbasedcachecoherenceprotocol.Amemorylocationmaybeinoneofthreestatesuncachednotcachedbyanyclustersharedinanunmodifiedstateinthecachesofoneormoreclustersordirtymodifiedinasinglecacheofsomecluster.Thedirectorykeepsthesummaryinformationforeachmemoryblock,specifyingitsstateandtheclustersthatarecachingit.TheDashmemorysystemcanbelogicallybrokenintofourlevelsofhierarchy,asillustratedinFigure3.Thefirstlevelistheprocessorscache.Thiscacheisdesignedtomatchtheprocessorspeedandsupportsnoopingfromthebus.Arequestthatcannotbeservicedbytheprocessorscacheissenttothesecondlevelinthehierarchy.thelocalcluster.Thislevelincludestheotherprocessorscacheswithintherequestingprocessorscluster.Ifthedataislocallycached,therequestcanbeservicedwithinthecluster.Otherwise,therequestissenttothehomeclusterlevel.Thehomelevelconsistsoftheclusterthatcontainsthedirectoryandphysicalmemoryforagivenmemoryaddress.Formanyaccessesforexample,mostprivatedatareferences.thelocalandhomeclusterarethesame,andthehierarchycollapsestothreelevels.Ingeneral,however,arequestwilltravelthroughtheinterconnectionnetworktothehomecluster.Thehomeclustercanusuallysatisfytherequestimmediately,butifthedirectoryentryisinadirtystate,orinsharedstatewhentherequestingprocessorrequestsexclusiveaccess,thefourthlevelmustalsobeaccessed.Theremoteclusterlevelforamemoryblockconsistsoftheclustersmarkedbythedirectoryasholdingacopyoftheblock.Toillustratethedirectoryprotocol,firstconsiderhowaprocessorreadtraversesthememoryhierarchyProcessorlevelIftherequestedlocationispresentintheprocessorscache,thecachesimplysuppliesthedata.Otherwise,therequestgoestothelocalclusterlevel.LocalclusterlevelIfthedataresideswithinoneoftheothercacheswithinthelocalcluster,thedataissupProcessorlevelProcessorcacheLocalclusterlevelOtherprocessorcacheswithinlocalclusterHomeclusterlevelIIDirectoryandmainmemoryassociatedwithagivenaddressIRemoteclusterlevelProcessorcachesin1remoteclustersFigure3.MemoryhierarchyofDash.pliedbythatcacheandnostatechangeisrequiredatthedirectorylevel.Iftherequestmustbesentbeyondthelocalclusterlevel,itgoesfirsttothehomeclustercorrespondingtothataddress.HomeclusterlevelThehomeclusterexaminesthedirectorystateofthememorylocationwhilesimultaneouslyfetchingtheblockfrommainmemory.Iftheblockisclean,thedataissenttotherequesterandthedirectoryisupdatedtoshowsharingbytherequester.Ifthelocationisdirty,therequestisforwardedtotheremoteclusterindicatedbythedirectory.RemoteclusterlevelThedirtyclusterreplieswithasharedcopyofthedata,whichissentdirectlytotherequester.Inaddition,asharingwritebackmessageissenttothehomeleveltoupdatemainmemoryandchangethedirectorystatetoindicatethattherequestingandremoteclusternowhavesharedcopiesofthedata.Havingthedirtyclusterresponddirectlytotherequester,asopposedtoroutingitthroughthehome.reducesthelatencyseenbytherequestingprocessor.NowconsiderthesequenceofoperationsthatoccurswhenalocationiswrittenProcessorlevelIfthelocationisdirtyinthewritingprocessorscache,thewritecancompleteimmediately.Otherwise,areadexclusiverequestisissuedonthelocalclustersbustoobtainexclusiveownershipofthelineandretrievetheremainingportionofthecacheline.LocalclusterlevelIfoneofthecacheswithintheclusteralreadyownsthecacheline,thenthereadexclusiverequestisservicedatthelocallevelbyacachetocachetransfer.Thisallowsprocessorswithinaclustertoalternatelymodifythesamememoryblockwithoutanyinterclusterinteraction.Ifnolocalcacheownstheblock,thenareadexclusiverequestissenttothehomecluster.HomeclusterlevelThehomeclustercanimmediatelysatisfyanownershiprequestforalocationthatisintheuncachedorsharedstate.Inaddition,ifablockisinthesharedstate,thenallcachedcopiesmustbeinvalidated.Thedirectoryindicatestheclustersthathavetheblockcached.Invalidationrequestsaresenttotheseclusterswhilethehomeconcurrentlysendsanexclusivedatareplytotherequestingcluster.Ifthedirectoryindicatesthattheblockisdirty,thenthereadexclusiverequestmustbeforwardedtothedirtycluster,asinthecaseofaread.RemoteclusterlevelIfthedirectoryhadindicatedthatthememoryblockwasshared,thentheremoteclustersreceiveaninvalidationrequesttoeliminatetheirsharedcopy.Uponreceivingtheinvalidation,theremoteclusterssendanacknowledgmenttotherequestingcluster.Ifthedirectoryhadindicatedadirtystate,thenthedirtyclusterreceivesareadexclusiverequest.Asinthecaseoftheread,theremoteclusterrespondsdirectlytotherequestingclusterandsendsadirtytransfermessagetothehomeindicatingthattherequestingclusternowholdstheblockexclusively.Whenthewritingclusterreceivesalltheinvalidationacknowledgmentsorthereplyfromthehomeordirtycluster,itisguaranteedthatallcopiesoftheolddatahavebeenpurgedfromthesystem.Iftheprocessordelayscompletingthewriteuntilallacknowledgmentsarereceived,thenthenewwritevaluewillbecomeavailabletoallotherprocessorsatthesametime.However,invalidationsinvolveroundtripmessagestomultipleclusters,resultinginpotentiallylongdelays.Higherprocessorutilizationcanbeobtainedbyallowingthewritetoproceedimmediatelyafterthe68COMPUTERownershipreplyisreceivedfromthehome.Unfortunately,thismayleadtoReleaseconsistencyprovidesa10to40inconsistencieswiththememorymodelassumedbytheprogrammer.ThenextsectiondescribeshowDashrelaxestheconstraintsonmemoryrequestorderpercentincreaseining,whilestillprovidingareasonableperformanceoversequentialconsistency.programmingmodeltotheuser.Memoryconsistency.Thememoryconsistencymodelsupportedbyanarchitecturedirectlyaffectstheamountofbufferingandpipeliningthatcantakeplaceamongmemoryrequests.Inaddition,ithasadirecteffectonthecomplexityoftheprogrammingmodelpresentedtotheuser.ThegoalinDashistoprovidesubstantialfreedomintheorderingamongmemoryrequests,whilestillprovidingareasonableprogrammingmodeltotheuser.Atoneendoftheconsistencyspectrumisthesequentialconsistencymode1,whichrequiresexecutionoftheparallelprogramtoappearasaninterleavingoftheexecutionoftheparallelprocessesonasequentialmachine.Sequentialconsistencycanbeguaranteedbyrequiringaprocessortocompleteonememoryrequestbeforeitissuesthenextrequest.4Sequentialconsistency,whileconceptuallyappealing,imposesalargeperformancepenaltyonmemoryaccesses.Formanyapplications,suchamodelistoostrict,andonecanmakedowithaweakernotionofconsistency.Asanexample,considerthecaseofaprocessorupdatingadatastructurewithinacriticalsection.Ifupdatingthestructurerequiresseveralwrites,eachwriteinasequentiallyconsistentsystemwillstalltheprocessoruntilallothercachedcopiesofthatlocationhavebeeninvalidated.Butthesestallsareunnecessaryastheprogrammerhasalreadymadesurethatnootherprocesscanrelyontheconsistencyofthatdatastructureuntilthecriticalsectionisexited.Ifthesynchronizationpointscanbeidentified,thenthememoryneedonlybeconsistentatthosepoints.Inparticular,Dashsupportstheuseofthereleaseconsistencymodel,whichonlyrequirestheoperationstohavecompletedbeforeacriticalsectionisreleasedthatis,alockisunlocked.Suchaschemehastwoadvantages.First,itprovidestheuserwithareasonableprogrammingmodel,sincetheprogrammerisassuredthatwhenthecriticalsectionisexited,allotherprocessorswillhaveaconsistentviewofthemodifieddatastructure.Second,itpermitsreadstobypasswritesandtheinvalidationsofdifferentwriteoperationstooverlap,resultinginlowerlatenciesforaccessesandhigheroverallperformance.Detailedsimulationstudiesforprocessorswithblockingreadshaveshownthatreleaseconsistencyprovidesa10to40percentincreaseinperformanceoversequentialconsistency.Thedisadvantageofthemodelisthattheprogrammerorcompilermustidentifyallsynchronizationaccesses.TheDashprototypesupportsthereleaseconsistencymodelinhardware.Sinceweusecommercialmicroprocessors,theprocessorstallsonreadoperationsuntilthereaddataisreturnedfromthecacheorlowerlevelsofthememoryhierarchy.Writeoperations,however,arenonblocking.Thereisawritebufferbetweenthefirstandsecondlevelcaches.Thewritebufferqueuesupthewriterequestsandissuestheminorder.Furthermore,theservicingofwriterequestsisoverlapped.Assoonasthecachereceivestheownershipanddatafortherequestedcacheline,thewritedataisremovedfromthewritebufferandwrittenintothecacheline.Thenextwriterequestcanbeservicedwhiletheinvalidationacknowledgmentsforthepreviouswriteoperationsfilterin.Thus,parallelismexistsattwolevelstheprocessorexecutesotherinstructionsandaccessesitsfirstlevelcachewhilewriteoperationsarepending,andinvalidationsofmultiplewriteoperationsareoverlapped.TheDashprototypealsoprovidesfenceoperationsthatstalltheprocessororwritebufferuntilpreviousoperationscomplete.Thesefenceoperationsallowsoftwaretoemulatemorestringentconsistencymodels.Memoryaccessoptimizations.Theuseofreleaseconsistencyhelpshidethelatencyofwriteoperations.However,sincetheprocessorstallsonreadoperations,itseestheentiredurationofallreadaccesses.Forapplicationsthatexhibitpoorcachebehaviororextensiveread/writesharing,thiscanleadtosignificantdelayswhiletheprocessorwaitsforremotecachemissestobefilled.TohelpwiththeseproblemsDashprovidesavarietyofprefetchandpipeliningoperations.Prefetchoperations.Aprefetchoperationisanexplicitnonblockingrequesttofetchdatabeforetheactualmemoryoperationisissued.Hopefully,bythetimetheprocessneedsthedata,itsvaluehasbeenbroughtclosertotheprocessor,hidingthelatencyoftheregularblockingread.Inaddition,nonblockingprefetchallowsthepipeliningofreadmisseswhenmultiplecacheblocksareprefetched.Asasimpleexampleofitsuse,aprocesswantingtoaccessarowofamatrixstoredinanotherclustersmemorycandosoefficientlybyfirstissuingprefetchreadsforallcacheblockscorrespondingtothatrow.Dashsprefetchoperationsarenonbindingandsoftwarecontrolled.Theprocessorissuesexplicitprefetchoperationsthatbringasharedorexclusivecopyofthememoryblockintotheprocessorscache.Notbindingthevalueatthetimeoftheprefetchisimportantinthatissuingtheprefetchdoesnotaffecttheconsistencymodelorforcethecompilertodoaconservativestaticdependencyanalysis.Thecoherenceprotocolkeepstheprefetchedcachelinecoherent.Ifanotherprocessorhappenstowritetothelocationbeforetheprefetchingprocessoraccessesthedata,thedatawillsimplybeinvalidated.Theprefetchwillberenderedineffective,buttheprogramwillexecutecorrectly.Supportforanexclusiveprefetchoperationaidscaseswheretheblockisfirstreadandthenupdated.Byfirstissuingtheexclusiveprefetch,theprocessoravoidsfirstobtainingasharedcopyandthenhavingtorerequestanexclusivecopyoftheblock.Studieshaveshownthat,forcertainapplications,theadditionofasmallnumberofprefetchinstructionscanincreaseprocessorutilizationbymorethanafactoroftwo.Updateanddeliveroperations.Insomeapplications,itmaynotbepossiblefortheconsumerprocesstoissueaprefetchearlyenoughtoeffectivelyhidethelatencyofmemory.Likewise,ifmultipleMarch199269consumersneedthesameitemofdata,thecommunicationtrafficcanbereducedifdataismulticasttoalltheconsumerssimultaneously.Therefore,Dashprovidesoperationsthatallowtheproducertosenddatadirectlytoconsumers.Therearetwowaysfortheproducingprocessortospecifytheconsumingprocessors.Theupdatewriteoperationsendsthenewdatadirectlytoallprocessorsthathavecachedthedata,whilethedeliveroperationsendsthedatatospecifiedclusters.Theupdatewriteprimitiveupdatesthevalueofallexistingcopiesofadataword.Usingthisprimitive,aprocessordoesnotneedtofirstacquireanexclusivecopyofthecacheline,whichwouldresultininvalidatingallothercopies.Rather,dataisdirectlywrittenintothehomememoryandallothercachesholdingacopyoftheline.Thesesemanticsareparticularlyusefulforeventsynchronization,suchasthereleaseeventforabarrier.Thedeliverinstructionexplicitlyspecifiesthedestinationclustersofthetransfer.Tousethisprimitive,theproducerfirstwritesintoitscacheusingnormal,invalidatingwriteoperations.Theproducerthenissuesadeliverinstruction,givingthedestinationclustersasabitvector.Acopyofthecachelineisthensenttothespecifiedclusters,andthedirectoryisupdatedtoindicatethatthevariousclustersnowsharethedata.Thisoperationisusefulincaseswhentheproducermakesmultiplewritestoablockbeforetheconsumerswillwantitorwhentheconsumersareunlikelytobecachingtheitematthetimeofthewrite.Supportforsynchronization.Theaccesspatternstolocationsusedforsynchronizationareoftendifferentfromthoseforothershareddata.Forexample,wheneverahighlycontendedlockisreleased,waitingnodesrushtograbthelock.Inthecaseofbarriers,manyprocessorsmustbesynchronizedandthenreleased.Suchactivityoftencauseshotspotsinthememorysystem.Consequently,synchronizationvariablesoftenwarrantspecialtreatment.Inadditiontoupdatewrites,Dashprovidestwoextensionstothecoherenceprotocolthatdirectlysupportsynchronizationobjects.Thefirstisqueuebasedlocks,andthesecondisfetchandincrementoperations.Mostcachecoherentarchitectureshandlelocksbyprovidinganatomictestsetinstructionandacachedtestandtestsetschemeforspinwaiting.Ideally,thesespinlocksshouldmeetthefollowingcriteriaminimumamountoftrafficgenerlowlatencyreleaseofawaitingprolowlatencyacquisitionofafreelock.atedwhilewaiting,cessor,andCachedtestsetschemesaremoderatelysuccessfulinsatisfyingthesecriteriaforlowcontentionlocks,butfailforhighcontentionlocks.Forexample,assumethereareNprocessorsspinningonalockvalueintheircaches.Whenthelockisreleased,allNcachevaluesareinvalidated,andNreadsaregeneratedtothememorysystem.Dependingonthetiming,itispossiblethatallNprocessorscomebacktodothetestsetonthelocationoncetheyrealizethelockisfree,resultinginfurtherinvalidationsandrereads.Suchascenarioproducesunnecessarytrafficandincreasesthelatencyinacquiringandreleasingalock.ThequeuebasedlocksinDashaddressthisproblembyusingthedirectorytoindicatewhichprocessorsarespinningonthelock.Whenthelockisreleased,oneofthewaitingclustersischosenatrandomandisgrantedthelock.Thegrantrequestinvalidatesonlythatclusterscachesandallowsoneprocessorwithinthatclustertoacquirethelockwithalocaloperation.Thisschemelowersboththetrafficandthelatencyinvolvedinreleasingaprocessorwaitingonalock.Informingonlyoneclusterofthereleasealsoeliminatesunnecessarytrafficandlatencythatwouldbeincurredifallwaitingprocessorswereallowedtocontend.Atimeoutmechanismonthelockgrantallowsthegranttobesenttoanotherclusterifthespinningprocesshasbeenswappedoutormigrated.ThequeuedonlockbitprimitivedescribedinGoodmanetal.issimilartoDashsqueuebasedlocks,butusespointersintheprocessorcachestomaintainthelistofthewaitingprocessors.Thefetchandincrementandfetchunddecrementprimitivesprovideatomicincrementanddecrementoperationsonuncachedmemorylocations.Thevaluereturnedbytheoperationsisthevaluebeforetheincrementordecrement.Theseoperationshavelowserializationandareusefulforimplementingseveralsynchronizationprimitivessuchasbarriers,distributedloops,andworkqueues.Theserializationoftheseoperationsissmallbecausetheyaredonedirectlyatthememorysite.Thelowserializationprovidedbythefetchandincrementoperationisespeciallyimportantwhenmanyprocessorswanttoincrementalocation,ashappenswhengettingthenextindexinadistributedloop.Thebenefitsoftheproposedoperationsbecomeapparentwhencontrastedwiththealternativeofusinganormalvariableprotectedbyalocktoachievetheatomicincrementanddecrement.Thealternativeresultsinsignificantlymoretraffic,longerlatency,andincreasedserialization.TheDashimplementationAhardwareprototypeoftheDasharchitectureiscurrentlyunderconstruction.Whilewehavedevelopedadetailedsoftwaresimulatorofthesystem,wefeelthatahardwareimplementationisneededtofullyunderstandtheissuesinthedesignofscalablecachecoherentmachines,toverifythefeasibilityofsuchdesigns,andtoprovideaplatformforstudyingrealapplicationsandsoftwarerunningonalargeensembleofprocessors.Tofocusoureffortonthenovelaspectsofthedesignandtospeedthecompletionofausablesystem,thebaseclusterhardwareusedintheprototypeisacommerciallyavailablebusbasedmultiprocessor.Whiletherearesomeconstraintsimposedbythegivenhardware,theprototypesatisfiesourprimarygoalsofscalablememorybandwidthandhighperformance.TheprototypeincludesmostofDashsarchitecturalfeaturessincemanyofthemcanonlybefullyevaluatedontheactualhardware.Thesystemalsoincludesdedicatedperformancemonitoringlogictoaidintheevaluation.Dashprototypecluster.TheprototypesystemusesaSiliconGraphicsPowerStation4D1340asthebasecluster.The4D1340systemconsistsoffourMipsR3000processorsandR3010floatingpointcoprocessorsrunningat33megahertz.EachR30001R3010combinationcanreachexecutionratesupto25VAXMIPSand10Mflops.EachCOMPUTER1rLfJLIIhinterfacememoryReplymeshIProcessorFirstlevelIandDcacheISecondlevelcachc1ITIinterfacememoryII\IInstructionDDataFigure4.Blockdiagramofa2x2Dashsystem.CPUcontainsa64kilobyteinstructioncacheanda64Kbytewritethroughdatacache.The64Kbytedatacacheinterfacestoa256Kbytesecondlevelwritebackcache.Theinterfaceconsistsofareadbufferandafourworddeepwritebuffer.Boththefirstandsecondlevelcachesaredirectmappedandsupport16bytelines.Thefirstlevelcachesrunsynchronouslytotheirassociated33MHzprocessorswhilethesecondlevelcachesrunsynchronoustothe16MHzmemorybus.Thesecondlevelprocessorcachesareresponsibleforbussnoopingandmaintainingcoherenceamongthecachesinthecluster.CoherenceismaintainedusinganIllinois,orMESImodified,exclusive,shared,invalid,protocol.ThemainadvantageofusingtheIllinoisprotocolinDashisthecachetocachetransfersspecifiedinit.Whiletheydolittletoreducethelatencyformissesscrvicedbylocalmemory.localcachetocachetransferscangreatlyreducethepenaltyforremotememorymisses.Thesetofprocessorcachesactsasaclustercacheforremotememory.ThememorybusMPbusofthe4D1340isasynchronousbusandconsistsofseparate32bitaddressand64bitdatabuses.TheMPbusispipelinedandsupportsmemorytocacheandcachetocachetransfersof16byteseveryfourbusclockswithalatencyofsixbusclocks.Thisresultsinamaximumbandwidthof64Mbytespersecond.WhiletheMPbusispipelined,itisnotasplittransactionbus.Tousethe4D1340inDash,wehavehadtomakeminormodificationstotheexistingsystemboardsanddesignapairofnewboardstosupportthedirectorymemoryandinterclusterinterface.Themainmodificationtotheexistingboardsistoaddabusretrysignalthatisusedwhenarequestrequiresservicefromaremotecluster.Thecentralbusarbiterhasalsobeenmodifiedtoacceptamaskfromthedirectory.Themaskholdsoffaprocessorsretryuntiltheremoterequesthasbeenserviced.Thiseffectivelycreatesasplittransactionbusprotocolforrequestsrequiringremoteservice.Thenewdirectorycontrollerboardscontainthedirectorymemory,theinterclustercoherencestatemachinesandbuffers,andalocalsectionoftheglobalinterconnectionnetwork.Theinterconnectionnetworkconsistsofapairofwormholeroutedmeshes,eachwith16bitwidechannels.Onemeshisdedicatedtotherequestmessageswhiletheotherhandlesreplies.Figure4showsablockdiagramoffourclustersconnectedtoforma2x2Dashsystem.SuchasystemcouldscaletosupporthundredsMarch199271