NPACIRocksTutorial.ppt

npacirockstutorial,npaciallhandsmeetingmarch18,2003,copyright2003ucregents,schedule,rocks101(45mins)introductiontorocksclusterarchitecture(hw&sw)rockssoftwareoverviewadvancedrocks(45mins)modifyingthedefaultconfigurationclustermonitoring(ucbsganglia)handsonlabs(90mins)breakoutintogroupsandbuildanx86clusterrunanmpijobcustomization(packages,monitoring)comparetoia64cluster,rocks101,copyright2003ucregents,makeclusterseasy,enableapplicationscientiststobuildandmanagetheirownresourceshardwarecostisnottheproblemsystemadministratorscostmoney,anddonotscalesoftwarecanreplacemuchoftheday-to-daygrindofsystemadministrationtrainthenextgenerationofusersonlooselycoupledparallelmachinescurrentprice-performanceleaderforhpcuserswillbereadyto“stepup”tonpaci(orother)resourceswhenneededrocksscalestotop500sizedresourcesexperimentonsmallclustersbuildyourownsupercomputerwiththesamesoftware!,copyright2003ucregents,past,rocks1.0releasedatsc2000goodstartonautomatingclusterbuildingincludedearlyprototype(writteninperl)ofgangliaresultofcollaborationbetweensdscanducbsmillenniumgroup.rocks2.xfullyautomatedinstallation(frontendandcomputenodes)programmablesoftwareconfigurationscs(singapore)firstexternalgrouptocontributesoftwarepatches,copyright2003ucregents,present,rocks2.3.2(todayslabispartofourbetatesting)firstsimultaneousx86andia64releaseprovenscalability#233oncurrenttop500list287-nodeproductionclusteratstanfordyoucanalsobuildsmallclustersimpactrocksclusterson6continentsnoantarcticayet.4largensfitrsusingrocksascoresoftwareinfrastructure,rocksregistrationpage(5daysold),/rocks-register,copyright2003ucregents,rocksinthecommercialworld,rocksclustervendorscraydellpromicrosystemsseepage79ofaprilslinuxjournalscs(insingapore)contributedpvfs,sgetorocksactiveontherocksmailinglisttrainingandsupportintelistrainingcustomersonrockscallidentisofferingsupportservices,copyright2003ucregents,promicrosystems,copyright2003ucregents,alternativeclustertoolkits,cplantdoeclusteringsoftwareopenmosixsinglesystemimageopenscekasetsartuniversity,thailandcompatiblewithrocksoscarakacluster-in-a-box,msc.linux“best-practicesofclustering”packagingeffort,notaclusterdistributionscyldbprocbasedsystem(ssiprocessspace)commercialscoreoldestclusteringefforttouchesonallaspectsofclusteringsoftware,copyright2003ucregents,whatsetsusapart,fullyautomatedclusterdeploymentgetandburnisocd(dvdforia64)imagefrombootfrontendwithcd/dvdfillout7configurationscreens(mostlyredhat)rebootfrontendmachineintegratecomputenodeswithinsert-ethersreadytogo!completeoutoftheboxsolutionwithrationaldefaultsettingsidenticalenvironmentforx86oria64,testimonialsandcomplaints,fromtherocks-discussmaillistandothersources,copyright2003ucregents,newuseronrocksmaillist,“imanagedtoinstallrockswithfivenodes.thenodeshaveasmallhd2.5gbeach,theclusterisinmyhomeonaprivatenetworkbehindalinuxboxfirewall.anditlookslikeeverythingisworkingfine.icanseeallthenodesandthefront-endinthegangliawebinterface.ibuiltitsoicanlearnmoreaboutclusters.andtotellthetruthihavenoideaonwhattodowithit,imeanwheretostart,howtouseit,whattouseitfor.”,copyright2003ucregents,powerusers,responsetopreviousposter“itssometimesscaryhoweasyitistoinstallrocks.”thiscoinedthephrase“rocks,scarytechnology”commentfromaclusterarchitect“youguysarethemastersofthebleedinobvious.”thisoneflatteredus,copyright2003ucregents,anothernewuser,“ivesetuparocksclusterthankstothebonesimpleinstallation.thanksformakingitsoeasy.thedrawback,becauseitwassoeasy,ididntlearnmuchaboutclustering.”,copyright2003ucregents,independentsurveyofclusteringsoftware,http:/heppc11.ft.uam.es/clusters/doccomparesrocks,oscarandothers“npacirocksistheeasiestsolutionfortheinstallationandmanagementofaclusterunderlinux.”“toinstallandmanagealinuxclusterunderoscarismoredifficultthanwithrocks.”“withoscaritisnecessarytohavesomeexperienceinlinuxsystemadministration,andsomeknowledgeofclusterarchitecture”ourgoalisto“makeclusterseasy”automatethesystemadministrationwhereverpossibleenablenon-clusterexpertstobuildclusters,copyright2003ucregents,andfinallyaweeklymessage,“yourdocumentationsucks.”guilty,butimprovingrocksnowinstallsusersguideoneverynewclustermailinglisthasseveralextremelyhelpfulusers,hardware,copyright2003ucregents,basicsystemarchitecture,copyright2003ucregents,whyx86clusters?,source:davepierce,sio,copyright2003ucregents,minimumcomponents,x86/ia64server,localharddrive,power,ethernet,copyright2003ucregents,optionalcomponents,myrinethigh-performancenetworknetwork-addressablepowerdistributionunitkeyboard/video/mousenetworknotrequirednon-commodityhowdoyoumanageyourmanagementnetwork?crashcartshavealowertco,software,copyright2003ucregents,clustersoftwarestack,copyright2003ucregents,commontoanycluster,copyright2003ucregents,redhat,stockredhat7.3w/updates(aw2.1foria64)linux2.4kernelnosupportforotherdistributionsredhatisthemarketleaderforlinuxintheusandbecomingsoineuropeexcellentsupportforautomatedinstallationscriptableinstallation(kickstart)verygoodhardwaredetection,copyright2003ucregents,batchsystems,portablebatchsystemmomdaemononeverynodeusedforjoblaunchingandhealthreportingserveronthefrontendonlyqueuedefinition,andaggregationofnodeinformationschedulerpoliciesforwhatjobtorunoutofwhichqueueatwhattimemauiisthecommonhpcschedulersge-sungridenginealternativetopbsintegratedintorocksbyscs(singapore)schedulermanagescarceresourcesclustersarecheapyoumightnotwantascheduler,copyright2003ucregents,communicationlayer,none“embarrassinglyparallel”socketsclient-servermodelpoint-to-pointcommunicationmpi-messagepassinginterfacemessagepassingstaticmodelofparticipantspvm-parallelvirtualmachinesmessagepassingforheterogeneousarchitecturesresourcecontrolandfaulttolerance,copyright2003ucregents,socketsarelowlevel,socketspoint-to-pointnmachines=(n2-n)/2connections1,3,6,10,15,mpi/pvmsharedvirtualchannelimplementationcouldbesocketseasiertoprogram,copyright2003ucregents,rocksclustersoftware,copyright2003ucregents,clusterstatemanagement,staticinformationnodeaddressesnodetypessite-specificconfigurationdynamicinformationcpuutilizationdiskutilizationwhichnodesareonline,copyright2003ucregents,clusterdatabase,copyright2003ucregents,ganglia,scalableclustermonitoringsystembasedonipmulti-castmattmassie,etalfromucbgmondaemononeverynodemulticastssystemstatelistenstootherdaemonsalldataisrepresentedinxmlgangliacommandlinepythoncodetoparsexmltoenglishgmetricextendsgangliacommandlinetomulticastsinglemetrics,copyright2003ucregents,gangliascreenshot,copyright2003ucregents,clustersoftwaremanagement,softwarepackagesrpmsstandardredhat(desktop)packagedsoftwareoryourownaddonsrocks-distmanagestherpmrepositorythisisthedistribution,softwareconfigurationtuningrpmsforclustersforyoursiteothercustomizationxmlkickstartprogrammaticsystembuildingscalable,copyright2003ucregents,buildingarocksdistribution,startwithredhataddupdates,rocks(andoptionalother)softwareaddkickstartprofilesmodifyredhatinstallationbootimageresultinginaredhatcompatiblerocksdistribution,copyright2003ucregents,kickstart,redhatskickstartmonolithicflatasciifilenomacrolanguagerequiresforkingbasedonsiteinformationandnodetype.rocksxmlkickstartdecomposeakickstartfileintonodesandagraphgraphspecifiesooframeworkeachnodespecifiesaserviceanditsconfigurationmacrosandsqlforsiteconfigurationdrivenfromwebcgiscript,copyright2003ucregents,samplenodefile,enablessheofhost*forwardx11yesforwardagentyeseofchmodo+rx/rootmkdir/root/.sshchmodo+rx/root/.ssh,copyright2003ucregents,samplegraphfile,defaultgraphfornpacirocks.,copyright2003ucregents,kickstartframework,copyright2003ucregents,appliances,laptop/desktopappliancesfinalclassesnodetypesdesktopisastandalonelaptopisastandalonepcmciacodere-useisgood,copyright2003ucregents,optionaldrivers,pvfsparallelvirtualfilesystemkernelmodulebuiltforallnodesinitialsupport(fullsupportinfutureversionofrocks)usermustdecidetoenablemyrinethighspeedandlowlatencyinterconnectgm/mpiforuserapplicationskernelmodulebuiltforallnodeswithmyrinetcards,copyright2003ucregents,yourclustersoftware,letsbuildacluster,copyright2003ucregents,hardwaresetup,grabtwonodesusethecross-overcabletoconnecteth0onthefrontendtoeth0onthecomputenodehookupamonitorandkeyboard,copyright2003ucregents,softwareinstallation,putthecdinthefrontendattheboot:prompt,type:frontendstartconfiguration,copyright2003ucregents,clusterinformation,copyright2003ucregents,partitionyourdisks,copyright2003ucregents,configureprivatenetwork,copyright2003ucregents,configurepublicnetwork,copyright2003ucregents,setyourhostname,copyright2003ucregents,settherootpassword,copyright2003ucregents,configurenis,copyright2003ucregents,andawaywego,advancedrocks,copyright2003ucregents,addcomputenodewithinsert-ethers,collecttheethernetmacaddressofclusternodesonlydoneonce,duringintegrationpopulatesclusterdatabase,copyright2003ucregents,discoveredappliance,copyright2003ucregents,54,remotelystartingreinstallationontwonodes,53,monitortheinstallwithekv,copyright2003ucregents,gangliagmetrics,howtodesignnewmetricsforgangliatotrackandgraph.calleduser-definedmetrics,orgmetrics.1.writegatheringfunctioninpython.2.pushtoallcomputenodes.3.allowgscheduletocallgathererrepeatedly.4.watchthegraphsgrow.,copyright2003ucregents,writemetricgathererinpython,writingagmetricgathereriseasyinpython:rockshasnativesupport.gscheduleisadaemonrunningbydefaultonallrocksnodes.willcallourgathererfunctionatregularintervals,withsomerandomization.wejustneedtoprovideanameandavalueforthemetric.,#sdscallhandsmeeting2003-advancedrockstutorial.#agmetric.#fromgmon.gmetricimportmetricfromstringimportcount,splitclasspackets(metric):publishesnumberofoutmulticastpacketsoneth0.dev=eth0def_init_(self):#publishvalueevery2secondsonaverage.metric._init_(self,2)defname(self):returnmcast-packets-in-%s%(self.dev)defvalue(self):dev=open(/proc/net/dev)forlineindev.readlines():ifnotcount(line,self.dev):continue#discarddev:fields=split(line,:)1values=split(fields)dev.close()returnint(values7),packets.py,copyright2003ucregents,pushnewgmetrictoallnodes,testonindividualnodeslogin(ssh)copypythonsourceto/opt/ganglia/lib/python/gmon/metrics/packaginguserpmforpermanentinstallation,computenodes,copyright2003ucregents,gscheduledaemon,therocksgscheduledaemonwillcallourpacketsmetricrepeatedly.gschedule-debugor#servicegschedulerestartwillpublishourmetricevery2seconds.gangliasgmetadwilldetectnewmetric,andgraphit.,gschedule,time,copyright2003ucregents,visualization,ournewgmetricwillbegraphedbygmetad.visibleinwebfrontendshostview.onlymetricswhicharenumericandvolatile(slopezero)aregraphed.gmetricrestrictions:totalmetricsize/tmp/ks.cfglookforcontrib-newin%packagessectionof/tmp/ks.cfg,copyright2003ucregents,reinstallcomputenodes,useshoot-nodetoreinstallcomputenodesinordertoapplythecontrib-newpackage#shoot-nodecompute-0-0,copyright2003ucregents,buildingacustomkernelrpm,pickacomputenode#sshcompute-0-0developonthisnode,doesnt“trash”theenvironmentonthefrontendcreateacustom.configfile.thiscanbedonefromscratchorbasedonacannedredhatconfigurationfile#cd/usr/src/linux-2.4#cpconfigs/kernel-2.4.18-i686-smp.config.config#vi.config,copyright2003ucregents,buildingacustomkernelrpm,buildakernelrpm#makerpmcopytheresultingkernelrpmtothefrontend:#scp/usr/src/redhat/rpms/i686/kernel.rpmfrontend-0:/home/install/contrib/rpms/public/i386rebuildthedistroonthefrontend#cd/home/install#rocks-distdist,copyright2003ucregents,computenodepartitioning,creates4gbrootpartitiononfirstdrivethispartitionisvolatile,thatis,whenthenodeisreinstalled,thispartitionisreformattedremainderoffirstdriveisputintoapartitioncalled“/state/partition1”foreachremainingdrives,onepartitioniscreatedperdriveandnamed“/state/partition2”,“/state/partition3”,etc.allpartitionslabeled“/state/partitionn”arenotreformattedonreboots.,copyright2003ucregents,example,rootdrive18gb/dev/sda1/4gb/dev/sda2/state/partition114gb,copyright2003ucregents,example,seconddrive36gb/dev/sdb1/state/partition236gb,copyright2003ucregents,example,thirddrive18gb/dev/sdc1/state/partition318gb,copyright2003ucregents,example,fourthdrive72gb/dev/sdd1/state/partition472gb,copyright2003ucregents,example,fifthdrive36gb/dev/sde1/state/partition536gb,copyright2003ucregents,example,sixthdrive181gb/dev/sdf1/state/partition6181gb,copyright2003ucregents,example,copyright2003ucregents,user-specifiedpartitioning,overridethedefaultpartitioningconfigurationfile:#cd/home/install/profiles/2.3.2/site-nodes#cpskeleton.xmlreplace-auto-partition.xmladdyourpartitionstoreplace-auto-partition.xml/-size5000-ondisksdaswap-size512-ondisksda/mydata-size1-grow-ondisksda,copyright2003ucregents,buildingyourowncdset,aftercustomizingyourclusterdistribution,makeisoimagestoexportitfirst,youmustmirrorthefullrocksrelease#cd/home/install#rocks-distmirroror,putinthesecond(thenthird)rockscdandexecute:#rocks-distcopycd,copyright2003ucregents,buildingyourowncdset,buildacdset#cd/home/install#rm-rfcdrom#rocks-dist-dist=cdromcdromthiscreatesacdsetunderthedirectory/home/install/cdrom#cd/home/install/cdrom/7.3/en/os,lab,copyright2003ucregents,thanks,inspiredbyalabconductedattacc,copyright2003ucregents,lab,building2-nodex86clustersgangliagridaddingusersrunningli