52-Memory Coherence in Shared Virtual Memory Systems.pdf

MemoryCoherenceinSharedVirtualMemorySystemsKAILlPrincetonUniversityandPAULHUDAKYaleUniversityThememorycoherenceproblemindesigningandimplementingasharedvirtualmemoryonlooselycoupledmultiprocessorsisstudiedindepth.Twoclassesofalgorithms,centralizedanddistributed,forsolvingtheproblemarepresented.AprototypesharedvirtualmemoryonanApolloringbasedonthesealgorithmshasbeenimplemented.Boththeoreticalandpracticalresultsshowthatthememorycoherenceproblemcanindeedbesolvedefficientlyonalooselycoupledmultiprocessor.CategoriesandSubjectDescriptors:C.2.1Computer-CommunicationNetworks:NetworkAr-chitectureandDesign-networkcommunications；C.2.4Computer-CommunicationNetworks:DistributedSystems-networkoperatingsystems；D.4.2OperatingSystems:StorageManage-ment-distributedmemories；uirtuolmemory；D.4.7OperatingSystems:OrganizationandDesign-distributedsystemsGeneralTerms:Algorithms,Design,Experimentation,Measurement,PerformanceAdditionalKeyWordsandPhrases:Looselycoupledmultiprocessors,memorycoherence,parallelprogramming,sharedvirtualmemory1.INTRODUCTIONThebenefitsofavirtualmemorygowithoutsaying；almosteveryhighperfor-mancesequentialcomputerinexistencetodayhasone.Infact,itishardtobelievethatlooselycoupledmultiprocessorswouldnotalsobenefitfromvirtualmemory.Onecaneasilyimaginehowvirtualmemorywouldbeincorporatedintoashared-memoryparallelmachinebecausethememoryhierarchyneednotbemuchdifferentfromthatofasequentialmachine.Onamultiprocessorinwhichthephysicalmemoryisdistributed,however,theimplementationisnotobvious.ThisresearchwassupportedinpartbyNationalScienceFoundationgrantsMCS-8302018,DCR-8106181,andCCR-8814265.ApreliminaryversionofthispaperappearedintheProceedingsofthe5thAnnualACMSymposiumonPrinciplesofDistributedComputing36.Authorsaddresses:K.Li,DepartmentofComputerScience,PrincetonUniversity,Princeton,NJ08544；P.Hudak,DepartmentofComputerScience,YaleUniversity,NewHaven,CT06520.Permissiontocopywithoutfeeallorpartofthismaterialisgrantedprovidedthatthecopiesarenotmadeordistributedfordirectcommercialadvantage,theACMcopyrightnoticeandthetitleofthepublicationanditsdateappear,andnoticeisgiventhatcopyingisbypermissionoftheAssociationforComputingMachinery.Tocopyotherwise,ortorepublish,requiresafeeand/orspecificpermission.01989ACM0734-2071/89/1100-0321$01.50ACMTransactionsonComputerSystems,Vol.7,No.4,November1989,Pages321-359.322K.LiandP.HudakThesharedvirtualmemorydescribedinthispaperprovidesavirtualaddressspacethatissharedamongallprocessorsinalooselycoupleddistributed-memorymultiprocessorsystem.Applicationprogramscanusethesharedvirtualmemoryjustastheydoatraditionalvirtualmemory,except,ofcourse,thatprocessescanrunondifferentprocessorsinparallel.Thesharedvirtualmemorynotonly“pages”databetweenphysicalmemoriesanddisks,asinaconventionalvirtualmemorysystem,butitalso“pages”databetweenthephysicalmemoriesoftheindividualprocessors.Thusdatacannaturallymigratebetweenprocessorsondemand.Furthermore,justasaconventionalvirtualmemoryswapsprocesses,sodoesthesharedvirtualmemory.Thusthesharedvirtualmemoryprovidesanaturalandefficientformofprocessmigrationbetweenprocessorsinadistributedsystem.Thisisquiteagainbecauseprocessmigrationisusuallyverydifficulttoimplement.Ineffect,processmigrationsubsumesremoteprocedurecalls.Themaindifficultyinbuildingasharedvirtualmemoryissolvingthememorycoherenceproblem.Thisproblemissimilartothatwhichariseswithmulticacheschemesforsharedmemorymultiprocessors,buttheyaredifferentinmanyways.Inthispaperweconcentrateonthememorycoherenceproblemforasharedvirtualmemory.Anumberofalgorithmsarepresented,analyzed,andcompared.AprototypesystemcalledIVYhasbeenimplementedonalocalareanetworkofApolloworkstations.Theexperimentalresultsofnontrivialparallelprogramsrunontheprototypeshowtheviabilityofasharedvirtualmemory.Thesuccessofthisimplementationsuggestsanoperatingmodeforsucharchi-tecturesinwhichparallelprogramscanexploitthetotalprocessingpowerandmemorycapabilitiesinafarmoreunifiedwaythanthetraditional“message-passing”approach.2.SHAREDVIRTUALMEMORYAsharedvirtualmemoryisasingleaddressspacesharedbyanumberofprocessors(Figure1).Anyprocessorcanaccessanymemorylocationintheaddressspacedirectly.Memorymappingmanagersimplementthemappingbetweenlocalmemoriesandthesharedvirtualmemoryaddressspace.Otherthanmapping,theirchiefresponsibilityistokeeptheaddressspacecoherentatalltimes；thatis,thevaluereturnedbyareadoperationisalwaysthesameasthevaluewrittenbythemostrecentwriteoperationtothesameaddress.Asharedvirtualmemoryaddressspaceispartitionedintopages.Pagesthataremarkedread-onlycanhavecopiesresidinginthephysicalmemoriesofmanyprocessorsatthesametime.Butapagemarkedwritecanresideinonlyoneprocessorsphysicalmemory.Thememorymappingmanagerviewsitslocalmemoryasalargecacheofthesharedvirtualmemoryaddressspaceforitsassociatedprocessor.Likethetraditionalvirtualmemory17,thesharedmemoryitselfexistsonlyvirtually.Amemoryreferencecausesapagefaultwhenthepagecontainingthememorylocationisnotinaprocessorscurrentphysicalmemory.Whenthishappens,thememorymappingmanagerretrievesthepagefromeitherdiskorthememoryofanotherprocessor.Ifthepageofthefaultingmemoryreferencehascopiesonotherprocessors,thenthememorymappingmanagermustdosomeworktokeepthememorycoherentandthencontinuethefaultinginstruction.ThispaperdiscussesbothcentralizedmanageralgorithmsandACMTransactionsonComputerSystems,Vol.7,No.4,November1989.MemoryCoherenceinSharedVirtualMemorySystemsl323SharedvirtualmemoryFig.1.Sharedvirtualmemorymapping.distributedmanageralgorithms,andinparticularshowsthataclassofdistributedmanageralgorithmscanretrievepagesefficientlywhilekeepingthememorycoherent.Ourmodelofaparallelprogramisasetofprocesses(orthreads)thatshareasinglevirtualaddressspace.Theseprocessesare“lightweight-theysharethesameaddressspace,andthusthecostofacontextswitch,processcreation,orprocessterminationissmall,say,ontheorderofafewprocedurecalls(RoyLevin,personalcommunication,1986).Oneofthekeygoalsofthesharedvirtualmemory,ofcourse,istoallowprocessesofaprogramtoexecuteondifferentprocessorsinparallel.Todoso,theappropriateprocessmanagerandmemoryallocationmanagermustbeintegratedproperlywiththememorymappingmanager.Theprocessmanagerandthememoryallocationmanageraredescribedelsewhere34.Werefertothewholesystemasashredvirtualmemorysystem.Theperformanceofparallelprogramsonasharedvirtualmemorysystemdependsprimarilyontwothings:thenumberofparallelprocessesandthedegreeofupdatingofshareddata(whichcreatescontentiononthecommunicationchannels).Sinceanyprocessorcanreferenceanypageinthesharedvirtualmemoryaddressspaceandmemorypagesaremovedandcopiedondemand,thesharedvirtualmemorysystemdoesnotexhibitpathologicalthrashingforun-shareddataorshareddatathatisread-only.Furthermore,updatingshareddatadoesnotnecessarilycausethrashingifaprogramexhibitslocalityofreference.Oneofthemainjustificationsfortraditionalvirtualmemoryisthatmemoryreferencesinsequentialprogramsgenerallyexhibitahighdegreeoflocality16,171.Althoughmemoryreferencesinparallelprogramsmaybehavedifferentlyfromthoseinsequentialones,asingleprocessisstillasequentialprogramandshouldexhibitahighdegreeoflocality.Contentionamongparallelprocessesforthesamepieceofdatadependsonthealgorithm,ofcourse,butacommongoalindesigningparallelalgorithmsistominimizesuchcontentionforoptimalperformance.Thereisalargebodyofliteraturerelatedtotheresearchofsharedvirtualmemory.Theclosestareasarevirtualmemoryandparallelcomputingonlooselycoupledmultiprocessors.Researchonvirtualmemorymanagementbeganinthe1960s15andhasbeenanimportanttopicinoperatingsystemdesigneversince.TheresearchACMTransactionsonComputerSystems,Vol.7,No.4,November1989.324lK.LiandP.Hudakfocusedonthedesignofvirtualmemorysystemsforuniprocessors.Anumberoftheearlysystemsusedmemorymappingtoprovideaccesstodifferentaddressspaces.TherepresentativesystemsareTenexandMultics5,131.Inthesesystems,processesindifferentaddressspacescansharedatastructuresinmappedmemorypages.Butthememorymappingdesignwasexclusivelyforuniprocessors.Spectorproposedaremotereference/remoteoperationmodel42inwhichamasterprocessonaprocessorperformsremotereferencesandaslaveprocessonanotherprocessorperformsremoteoperations.Usingprocessornamesaspartoftheaddressinremotereferenceprimitives,thismodelallowsalooselycoupledmultiprocessortobehaveinawaysimilartoCM*24,291orButterfly6inwhichasharedmemoryisbuiltfromlocalphysicalmemoriesinastaticmanner.Althoughimplementingremotememoryreferenceprimitivesinmicrocodecangreatlyimproveefficiency,thecostofaccessingaremotememorylocationisstillseveralordersofmagnitudemoreexpensivethanalocalmemoryreference.Themodelisusefulfordatatransferindistributedcomputing,butitisunsuitableforparallelcomputing.Amongthedistributedoperatingsystemsforlooselycoupledmultiprocessors,ApolloAegis2,32,331andAccent20,381havehadastronginfluenceontheintegrationofvirtualmemoryandinterprocesscommunication.BothAegisandAccentpermitmappedaccesstodataobjectsthatcanbelocatedanywhereinadistributedsystem.Bothofthemviewphysicalmemoryasacacheofvirtualstorage.Aegisusesmappedreadandwritememoryasitsfundamentalcommu-nicationparadigm.Accenthasasimilarfacilitycalledcopy-on-writeandamechanismthatallowsprocessestopassdatabyvalue.Thedatasharingbetweenprocessesinthesesystemsislimitedattheobjectlevel；thesystemdesignsarefordistributedcomputingratherthanparallelcomputing.Realisticparallelcomputingworkonlooselycoupledmultiprocessorshasbeenlimited.Muchworkhasfocusedonmessagepassingll,19,391.Itispossibletogainlargespeedupsoverauniprocessorbymessagepassing,butprogrammingapplicationsaredifficult111.Furthermore,asmentionedabove,messagepassinghasdifficultiesinpassingcomplicateddatastructures.Anotherdirectionhasbeentouseasetofprimitives,availabletotheprogram-merinthesourcelanguage,toaccessaglobaldataspaceforstoringshareddatastructures8,111.Thechiefproblemwithsuchanapproachistheusersneedtocontroltheglobaldataspaceexplicitly,whichcanbecomeespeciallycomplexwhenpassinglargedatastructuresorwhenattemptingprocessmigration.Inasharedvirtualmemorysuchaswepropose,noexplicitdatamovementisrequired(ithappensimplicitlyuponmemoryreference),andcomplexdataismovedaseasilyassimpledata.Anotherseriousproblemwiththeexplicitglobaldataspaceapproachisthatefficiencyisimpairedevenforlocaldatasinceuseofaprimitiveimpliesatleasttheoverheadofaprocedurecall.Thisproblembecomesespeciallyacuteifoneoftheprimitiveoperationsoccursinaninnerloop,inwhichcaseexecutionononeprocessorismuchslowerthanthatofthebestsequentialprogram,thatis,oneinwhichtheoperationisreplacedwithastandardmemoryreference.Incontrast,whenusingoursharedvirtualmemory,theinnerloopwouldlookjustthesameasitssequentialversion,andthustheoverheadforaccessinglocaldatawouldbeexactlythecostofastandardmemoryreference.ACMTransactionsonComputerSystems,Vol.7,No.4,November1989.MemoryCoherenceinSharedVirtualMemorySystemsl325Thepointbeingthat,oncethepagesholdingaglobaldatastructurearepagedin,themechanismforaccessingthedatastructureispreciselythesameasonauniprocessor.Theconceptofasharedvirtualmemoryforlooselycoupledmultiprocessorswasfirstproposedin36andelaboratedinthePh.D.dissertation34.Detailsofthefirstimplementation,IVY,onanetworkofworkstationswasreportedin34and35.Onthebasisofthisearlywork,asharedvirtualmemorysystemwaslaterdesignedfortheLotusoperatingsystemkernel21.Mostrecently,theconcepthasbeenappliedtoalarge-scaleinterconnectionnetworkbasedshared-memorymultiprocessor121andalarge-scalehypercubemultiprocessor37.Otherrelatedworkincludessoftwarecachesandanalysisofmemoryreferences.TheVMPprojectatStanfordimplementsasoftwarevirtualaddressedcachelotoprovidemulticomputerswithacoherentsharedmemoryspace.Theirinitialexperienceshowsthatacachelinesizecanbeaslargeas128or256byteswithoutperformancedegradation.Thecacheconsistencyprotocolissimilartothedynamicdistributedmanageralgorithmforsharedvirtualmemoryinthispaperanditsprelimmaryversion34.Finally,techniquesforanalyzingmemoryreferencesofparallelprograms18,451maybeapplicabletoanalyzingthebehaviorsofparallelprogramsusingasharedvirtualmemorysystem.3.MEMORYCOHERENCEPROBLEMAmemoryiscoherentifthevaluereturnedbyareadoperationisalwaysthesameasthevaluewrittenbythemostrecentwriteoperationtothesameaddress.Anarchitecturewithonememoryaccesspathshouldhavenocoherenceproblem.Asingleaccesspath,however,maynotsatisfytodaysdemandforhighperfor-mance.Thememorycoherenceproblemwasfirstencounteredwhencachesappearedinuniprocessors(see40forasurvey)andhasbecomemorecompli-catedwiththeintroductionof“multicaches”forsharedmemoriesonmultipro-cessors9,23,25,31,43,46andChuckThacher,personalcommunication,19841.Thememorycoherenceprobleminasharedvirtualmemorysystemdiffers,however,fromthatinmulticachesystems.Amulticachemultiprocessorusuallyhasanumberofprocessorssharingaphysicalmemorythroughtheirprivatecaches.Sincethesizeofacacheisrelativelysmallandthebusconnectingittothesharedmemoryisrelativelyfast,asophisticatedcoherenceprotocolisusuallyimplementedinthemulticachehardwaresuchthatthetimedelayofconflictingwritestoamemorylocationissmall.Ontheotherhand,asharedvirtualmemoryonalooselycoupledmultiprocessorhasnophysicallysharedmemory,andthecommunicationcostbetweenprocessorsisnontrivial.Thusconflictsarenotlikelytobesolvedwithnegligibledelay,andtheyresemblemuchmorea“pagefault”inatraditionalvirtualmemorysystem.Therearetwodesignchoicesthatgreatlyinfluencetheimplementationofasharedvirtualmemory:thegranularityofthememoryunits(i.e.,the“pagesize”)andthestrategyformaintainingcoherence.Thesetwodesignissuesarestudiedinthenexttwosubsections.3.1GranularityInatypicallooselycoupledmultiprocessor,sendinglargepacketsofdata(sayonethousandbytes)isnotmuchmoreexpensivethansendingsmallones(sayACMTransactionsonComputerSystems,Vol.7,No.4,November1989.326lK.LiandP.Hudaklessthantenbytes)41.ThissimilP&yincostisusuallyduetothesoftwareprotocolsandoverheadofthevirtualmemorylayeroftheoperatingsystem.Iftheseoverheadsareacceptable,relativelylargememoryunitsarepossibleinasharedvirtualmemory.Ontheotherhand,thelargerthememoryunit,thegreaterthechanceforcontention.Detailedknowledgeofaparticularimplemen-tationmightallowtheclientsprogrammertominimizecontentionbyarrangingconcurrentmemoryaccessestolocationsindifferentmemoryunits.EitherclientsorthesharedVirtualmemorystorageallocatormaytrytoemploysuchstrategies,butthismayintroduceinefficientuseofmemory.So,thepossibilityofcontentionindicatestheneedforrelativelysmallmemoryunits.Asuitablecompromiseingranularityisthetypicalpageasusedinconventionalvirtualmemoryimplementations,whichvaryinsizeontodayscomputersfrom256bytesto8Kbytes.Ourexperienceindicatesthatapagesizeofabout1Kbytesissuitablewithrespecttocontention,andasmentionedaboveshouldnotimposeunduecommunicationsoverhead.Weexpectthatsmallerpagesizes(perhapsaslowas256bytes)workwellalso,butwearenotasconfidentaboutlargerpagesizes,duetothecontentionproblem.Therightsizeisclearlyapplicationdependent,however,andwesimply&onothavetheimplementationexperiencetosaywhatsizeisbestforasufficientlybroadrangeofparallelprograms.Inanycase,choosingapagesizeconsistentwiththatusedinconven-tionalvirtualmemoryimplementationshastheadvantageofallowingonetouseexistingpagefaultschemes.Inparticular,onecanusetheprotectionmechanismsinahardwareMemoryManagementUnit(MMU)thatallowsingleinstructionstotriggerpagefaultsandtotrapappropriatefaulthandlers.Aprogramcansettheaccessrightatothepagesinsuchawaythatmemoryaccessesthatcouldviolatememorycoherencecauseapagefault；thusthememorycoherenceproblemcanbesolvedinamodularwayinthepagefaulthandlersandtheirservers；3.2MemoryCoherenceStrategiesItishelpfultofirstconsiderthespectrumofchoicesonehasforsolvingthememorycoherenceproblem.Thesechoicescanbeclassifiedbythewayinwhichonedealswithpagesynchronizationandpageownrship,tieshowninTableI.PageSynchronization.Therearetwobasicapproachestopagesynchroniza-tion:invalidationandwrite-broadcast.Intheinvalidationapproach,thereisonlyoneownerprocessorforeachpage.Thisprocessorhaseitherwriteorreadaccesstothepage.IfaprocessorQhasawritefaulttoapagep,itsfaulthandlerthen-invalidatesallcopiesofp,-changestheaccessofptowrite,-movesacopyofptoQifQdoesnothaveonealready,and-returnstothefaultinginstruction.Afterreturning,processorQ“owns”pagepandcanproceedwiththewriteoperationandotherreadorwriteoperationsuntilthepageownershipisrelin-quishedtosomeotherprocessor.ProcessorQ,ofcourse,doesnotneedtomovethecopyofthepageifitownsthepageforreading.IfaprocessorQhasareadACMTransactionsonComputerSystems,Vol.7,No.4,November1989.MemoryCoherenceinSharedVirtualMemorySystemsl327TableI.SpectrumofSolutionstotheMemoryCoherenceProblemPageownershipstrategyPagesynchronizationmethodDynamicFixedCentralizedmanagerDistributedmanagerFixedDynamicInvalidationNotallowedOkayGoodGoodWrite-broadcastVeryexpensiveVeryexpensiveVeryexpensiveVeryexpensivefaulttoapagep,thefaulthandlerthen-changestheaccessofptoreadontheprocessorthathaswriteaccesstop,-movesacopyofptoQandsetstheaccessofptoread,and-returnstothefaultinginstruction.Afterreturning,processorQcanproceedwiththereadoperationandotherreadoperationstothispageinthesamewaythatnormallocalmemorydoesuntilpisrelinquishedtosomeoneelse.Inthewrite-broadcastapproach,aprocessortreatsareadfaultjustasitdoesintheinvalidationapproach.However,ifaprocessorhasawritefault,thefaulthandlerthen-writestoallcopiesofthepage,and-returnstothefaultinginstruction.Themainproblemswiththisapproachisthatitrequiresspecialhardwaresupport.Everywritetoasharedpageneedstogenerateafaultonthewritingprocessorandupdateallcopiesbecausethephilosophyofasharedvirtualmemoryrequiresthatpagesbesharedfreely.Topreventtheprocessorfromhavingthesamepagefaultagainwhenreturningtothefaultinginstruction,thehardwaremustbeabletoskipthefaultedwritecycle.Wedonotknowofanyexistinghardwarewiththisfunctionality.Thetheoreticalanalysison“snoopycache”coherence30suggeststhatcombiningtheinvalidationapproachwiththewrite-broadcastapproachmaybeabettersolution.However,whetherthisapproachcanapplytothesharedvirtualmemoryisanopenproblembecausetheoverheadofawritefaultismuchmorethanawriteonasnoopycachebus.Sincethealgorithmsusingwrite-broadcastdonotseempracticalforlooselycoupledmultiprocessors,theyarenotconsideredfurtherinthispaper.PageOwnership.Theownershipofapagecanbefixedordynamic.Inthefixedownershipapproach,apageisalwaysownedbythesameprocessor.Otherprocessorsarenevergivenfullwriteaccesstothepage；rathertheymustnegotiatewiththeowningprocessorandmustgenerateawritefaulteverytimetheyneedtoupdatethepage.Aswiththewrite-broadcastapproach,fixedpageownershipACMTransactionsonComputerSystems,Vol.7,No.4,November1989.328lK.LiandP.Hudakisanexpensivesolutionforexistinglooselycoupledmultiprocessors.Further-more,itconstrainsdesiredmodesofparallelcomputation.Thusweonlyconsiderdynamicownershipstrategies,asindicatedinTableI.Thestrategiesformaintainingdynamicpageownershipcanbesubdividedintotwoclasses:centralizedanddistributed.Distributedmanagerscanbefurtherclassifiedaseitherfixedordynamic,referringtothedistributionofownershipdata.TheresultingcombinationsofstrategiesareshowninTableI,wherewehavemarkedas“veryexpensive”or“notallowed”allcombinationsinvolvingwrite-broadcastsynchronizationorfixedpageownership.Thispaperonlyconsiderstheremainingchoices:algorithmsbasedoninvalidationusingeitheracentralizedmanager,afixeddistributedmanager,oradynamicdistributedmanager.3.3PageTable,Locking,andinvalidationAllofthealgorithmsforsolvingthememorycoherenceprobleminthispaperaredescribedbyusingpagefaulthandlers,theirservers,andthedatastructureonwhichtheyoperate.Thedatastructuresindifferentalgorithmsmaybedifferent,buttheyhaveatleastthefollowinginformationabouteachpage:-access:indicatestheaccessibilitytothepage,-copyset:containstheprocessornumbersthathavereadcopiesofthepage,and-lock:synchronizesmultiplepagefaultsbydifferentprocessesonthesameprocessorandsynchronizesremotepagerequests.Followinguniprocessorvirtualmemoryconvention,thisdatastructureiscalledapagetable.Everyprocessorusuallyhasapagetableonit,butthesamepageentryindifferentpagetablesmaybedifferent.Therearetwoprimitivesoperatingonthelockfieldinthepagetable:/ock(PTablep.lock):LOOPIFtest-and-setthelockbitTHENEXIT；IFfailTHENqueuethisprocess；un/ock(PTablepJ.lock):clearthelockbit；IFaprocessiswaitingonthelockTHENresumetheprocess；Thesetwoprimitivesareusedtosynchronizemultiplepagefaultrequestsonthesameprocessorordifferentprocessors.Anotherprimitivethatweuseinmemorycoherencealgorithmsisinvalidate.Thereareatleastthreewaystoinvalidatethecopiesofapage:individual,broadcast,andmulticast.Theindividualinvalidationisjustasimpleloop:Invalidate:Invalidate(p,copy-set)FORiincopy-setDOsendaninvalidationrequesttoprocessori；Broadcastormulticastinvalidationdoesnotneedacopyset；eachjustrequiresasimplebroadcastmessage.ACMTransactionsonComputerSystems,Vol.7,No.4,November1989.MemoryCoherenceinSharedVirtualMemorySystemsl329Theserveroftheinvalidationoperationissimple:Invalidateserver:PTablep.access:=nil；Althoughtherearemanywaystoimplementremoteoperations,itisreasonabletoassumethatanyremoteop