GOOGLE集群计算之Hadoop技术概要.ppt

SpinnakerLabs Inc GoogleClusterComputingFacultyTrainingWorkshop ModuleV HadoopTechnicalReview SpinnakerLabs Inc Overview HadoopTechnicalWalkthroughHDFSDatabasesUsingHadoopinanAcademicEnvironmentPerformancetipsandothertools YouSay tomato SpinnakerLabs Inc SomeMapReduceTerminology Job A fullprogram anexecutionofaMapperandReduceracrossadatasetTask AnexecutionofaMapperoraReduceronasliceofdataa k a Task In Progress TIP TaskAttempt Aparticularinstanceofanattempttoexecuteataskonamachine SpinnakerLabs Inc TerminologyExample Running WordCount across20filesisonejob20filestobemappedimply20maptasks somenumberofreducetasksAtleast20maptaskattemptswillbeperformed moreifamachinecrashes etc SpinnakerLabs Inc TaskAttempts Aparticulartaskwillbeattemptedatleastonce possiblymoretimesifitcrashesIfthesameinputcausescrashesoverandover thatinputwilleventuallybeabandonedMultipleattemptsatonetaskmayoccurinparallelwithspeculativeexecutionturnedonTaskIDfromTaskInProgressisnotauniqueidentifier don tuseitthatway SpinnakerLabs Inc MapReduce HighLevel SpinnakerLabs Inc Node to NodeCommunication HadoopusesitsownRPCprotocolAllcommunicationbeginsinslavenodesPreventscircular waitdeadlockSlavesperiodicallypollfor status messageClassesmustprovideexplicitserialization SpinnakerLabs Inc Nodes Trackers Tasks MasternoderunsJobTrackerinstance whichacceptsJobrequestsfromclientsTaskTrackerinstancesrunonslavenodesTaskTrackerforksseparateJavaprocessfortaskinstances SpinnakerLabs Inc JobDistribution MapReduceprogramsarecontainedinaJava jar file anXMLfilecontainingserializedprogramconfigurationoptionsRunningaMapReducejobplacesthesefilesintotheHDFSandnotifiesTaskTrackerswheretoretrievetherelevantprogramcode Where sthedatadistribution SpinnakerLabs Inc DataDistribution ImplicitindesignofMapReduce Allmappersareequivalent somapwhateverdataislocaltoaparticularnodeinHDFSIflotsofdatadoeshappentopileuponthesamenode nearbynodeswillmapinsteadDatatransferishandledimplicitlybyHDFS SpinnakerLabs Inc ConfiguringWithJobConf MRProgramshavemanyconfigurableoptionsJobConfobjectshold key value componentsmappingString ae g mapred map tasks 20JobConfisserializedanddistributedbeforerunningthejobObjectsimplementingJobConfigurablecanretrieveelementsfromaJobConf SpinnakerLabs Inc WhatHappensInMapReduce DepthFirst SpinnakerLabs Inc JobLaunchProcess Client ClientprogramcreatesaJobConfIdentifyclassesimplementingMapperandReducerinterfacesJobConf setMapperClass setReducerClass Specifyinputs outputsJobConf setInputPath setOutputPath Optionally otheroptionstoo JobConf setNumReduceTasks JobConf setOutputFormat SpinnakerLabs Inc JobLaunchProcess JobClient PassJobConftoJobClient runJob orsubmitJob runJob blocks submitJob doesnotJobClient DeterminesproperdivisionofinputintoInputSplitsSendsjobdatatomasterJobTrackerserver SpinnakerLabs Inc JobLaunchProcess JobTracker JobTracker InsertsjarandJobConf serializedtoXML insharedlocationPostsaJobInProgresstoitsrunqueue SpinnakerLabs Inc JobLaunchProcess TaskTracker TaskTrackersrunningonslavenodesperiodicallyqueryJobTrackerforworkRetrievejob specificjarandconfigLaunchtaskinseparateinstanceofJavamain isprovidedbyHadoop SpinnakerLabs Inc JobLaunchProcess Task TaskTracker Child main SetsupthechildTaskInProgressattemptReadsXMLconfigurationConnectsbacktonecessaryMapReducecomponentsviaRPCUsesTaskRunnertolaunchuserprocess SpinnakerLabs Inc JobLaunchProcess TaskRunner TaskRunner MapTaskRunner MapRunnerworkinadaisy chaintolaunchyourMapperTaskknowsaheadoftimewhichInputSplitsitshouldbemappingCallsMapperonceforeachrecordretrievedfromtheInputSplitRunningtheReducerismuchthesame SpinnakerLabs Inc CreatingtheMapper YouprovidetheinstanceofMapperShouldextendMapReduceBaseOneinstanceofyourMapperisinitializedbytheMapTaskRunnerforaTaskInProgressExistsinseparateprocessfromallotherinstancesofMapper nodatasharing SpinnakerLabs Inc Mapper voidmap WritableComparablekey Writablevalue OutputCollectoroutput Reporterreporter SpinnakerLabs Inc WhatisWritable Hadoopdefinesitsown box classesforstrings Text integers IntWritable etc AllvaluesareinstancesofWritableAllkeysareinstancesofWritableComparable SpinnakerLabs Inc WritingForCacheCoherency while moreinputexists myIntermediate newintermediate input myIntermediate process exportoutputs SpinnakerLabs Inc WritingForCacheCoherency myIntermediate newintermediate junk while moreinputexists myIntermediate setupState input myIntermediate process exportoutputs SpinnakerLabs Inc WritingForCacheCoherency RunningtheGCtakestimeReusinglocationsallowsbettercacheusageSpeedupcanbeasmuchastwo foldAllserializabletypesmustbeWritableanyway somakeuseoftheinterface GettingDataToTheMapper SpinnakerLabs Inc ReadingData DatasetsarespecifiedbyInputFormatsDefinesinputdata e g adirectory IdentifiespartitionsofthedatathatformanInputSplitFactoryforRecordReaderobjectstoextract k v recordsfromtheinputsource SpinnakerLabs Inc FileInputFormatandFriends TextInputFormat Treatseach n terminatedlineofafileasavalueKeyValueTextInputFormat Maps n terminatedtextlinesof kSEPv SequenceFileInputFormat Binaryfileof k v pairswithsomeadd lmetadataSequenceFileAsTextInputFormat Same butmaps k toString v toString SpinnakerLabs Inc FilteringFileInputs FileInputFormatwillreadallfilesoutofaspecifieddirectoryandsendthemtothemapperDelegatesfilteringthisfilelisttoamethodsubclassesmayoverridee g Createyourown xyzFileInputFormat toread xyzfromdirectorylist SpinnakerLabs Inc RecordReaders EachInputFormatprovidesitsownRecordReaderimplementationProvides unused capabilitymultiplexingLineRecordReader ReadsalinefromatextfileKeyValueRecordReader UsedbyKeyValueTextInputFormat SpinnakerLabs Inc InputSplitSize FileInputFormatwilldividelargefilesintochunksExactsizecontrolledbymapred min split sizeRecordReadersreceivefile offset andlengthofchunkCustomInputFormatimplementationsmayoverridesplitsize e g NeverChunkFile SpinnakerLabs Inc SendingDataToReducers MapfunctionreceivesOutputCollectorobjectOutputCollector collect takes k v elementsAny WritableComparable Writable canbeused SpinnakerLabs Inc WritableComparator ComparesWritableComparabledataWillcallWritableCpare CanprovidefastpathforserializeddataJobConf setOutputValueGroupingComparator SpinnakerLabs Inc SendingDataToTheClient ReporterobjectsenttoMapperallowssimpleasynchronousfeedbackincrCounter Enumkey longamount setStatus Stringmsg Allowsself identificationofinputInputSplitgetInputSplit PartitionAndShuffle SpinnakerLabs Inc Partitioner intgetPartition key val numPartitions OutputsthepartitionnumberforagivenkeyOnepartition valuessenttooneReducetaskHashPartitionerusedbydefaultUseskey hashCode toreturnpartitionnumJobConfsetsPartitionerimplementation SpinnakerLabs Inc Reduction reduce WritableComparablekey Iteratorvalues OutputCollectoroutput Reporterreporter Keys valuessenttoonepartitionallgotothesamereducetaskCallsaresortedbykey earlier keysarereducedandoutputbefore later keys SpinnakerLabs Inc Finally WritingTheOutput SpinnakerLabs Inc OutputFormat AnalogoustoInputFormatTextOutputFormat Writes keyval n stringstooutputfileSequenceFileOutputFormat Usesabinaryformattopack k v pairsNullOutputFormat Discardsoutput SpinnakerLabs Inc HDFS SpinnakerLabs Inc HDFSLimitations Almost GFSNofileupdateoptions recordappend etc allfilesarewrite onceDoesnotimplementdemandreplicationDesignedforstreamingRandomseeksdevastateperformance SpinnakerLabs Inc NameNode Head interfacetoHDFSclusterRecordsallglobalmetadata SpinnakerLabs Inc SecondaryNameNode NotafailoverNameNode Recordsmetadatasnapshotsfrom real NameNodeCanmergeupdatelogsinflightCanuploadsnapshotbacktoprimary SpinnakerLabs Inc NameNodeDeath NonewrequestscanbeservedwhileNameNodeisdownSecondarywillnotfailoverasnewprimarySowhyhaveasecondaryatall SpinnakerLabs Inc NameNodeDeath cont d IfNameNodediesfromsoftwareglitch justrebootButifmachineishosed metadataforclusterisirretrievable SpinnakerLabs Inc BringingtheClusterBack IforiginalNameNodecanberestored secondarycanre establishthemostcurrentmetadatasnapshotIfnot createanewNameNode usesecondarytocopymetadatatonewprimary restartwholecluster Isthereanotherway SpinnakerLabs Inc KeepingtheClusterUp Problem DataNodes fix theaddressoftheNameNodeinmemory can tswitchinflightSolution BringnewNameNodeup butuseDNStomakeclusterbelieveit stheoriginaloneSecondarycanbethe new one SpinnakerLabs Inc FurtherReliabilityMeasures NamenodecanoutputmultiplecopiesofmetadatafilestodifferentdirectoriesIncludinganNFSmountedoneMaydegradeperformance watchforNFSlocks SpinnakerLabs Inc Databases SpinnakerLabs Inc LifeAfterGFS StraightGFSfilesarenottheonlystorageoptionHBase ontopofGFS providescolumn orientedstoragemySQLandotherdbenginesstillrelevant SpinnakerLabs Inc HBase CaninterfacedirectlywithHadoopProvidesitsownInput andOutputFormatclasses sendsrowsdirectlytomapper receivesnewrowsfromreducer Butmightnotbereadyforclassroomuse leaststablecomponent SpinnakerLabs Inc MySQLClustering MySQLdatabasecanbeshardedonmultipleserversForfastIO usesamemachinesasHadoopTablescanbesplitacrossmachinesbyrowkeyrangeMultiplereplicascanservesametable SpinnakerLabs Inc Sharding HadoopPartitioners Forbestperformance ReducershouldgostraighttolocalmysqlinstanceGetalldataintherightmachineinonecopyImplementcustomPartitionertoensureparticularkeyrangegoestomysql awareReducer SpinnakerLabs Inc AcademicHadoopRequirements SpinnakerLabs Inc ServerProfile UWcluster 40nodes 80processorstotal2GBram processor24TBrawstoragespace 8TBreplicated OnenodereservedforJobTracker NameNodeTwomorewouldn tcooperate Butstillvastlyoverpowered SpinnakerLabs Inc Setup Maintenance TookabouttwodaystosetupandconfigureMostlyhardware relatedissuesHadoopsetupwasonlyacouplehoursMaintenance onlyafewhours weekMostlyrebootingtheclusterwhenjobsgotstuck SpinnakerLabs Inc TotalUsage About15 000CPU hoursconsumedby20students Outof130 000availableoverquarterAverageloadisabout12 SpinnakerLabs Inc Analyzingstudentusagepatterns SpinnakerLabs Inc NotQuitetheWholeStory Realistically studentsdidmostworkveryclosetodeadlineClustersatunusedforafewdays followedbyoverloadingfortwodaysstraight SpinnakerLabs Inc Analyzingstudentusagepatterns Lesson ResourcedemandsareNOTconstant SpinnakerLabs Inc HadoopJobScheduling FIFOqueuematchesincomingjobstoavailablenodesNonotionoffairnessNeverswitchesoutrunningjobRun awaytaskscouldstarveotherstudentjobs SpinnakerLabs Inc HadoopSecurity Butonthebright side NosecuritysystemforjobsAnyonecanstartajob buttheycanalsocancelotherjobsRealistically studentsdidnotcancelotherstudentjobs evenwhentheyshould SpinnakerLabs Inc HadoopSecurity TheDarkSide NopermissionsinHDFSeitherJustnowaddedin0 16OnestudentdeletedthecommondatasetforaprojectEmailsubject Oops Nostudentscouldtesttheircodeuntildatasetrestoredfrombackup SpinnakerLabs Inc JobSchedulingLessons Gettingstudentsto playnice ishardNoincentiveJustplainbad buggycodeClustercontentioncausedproblemsatdeadlinesWorkingroupsStaggerdeadlines SpinnakerLabs Inc AnotherPossibility AmazonEC2provideson demandserversMaybeabletohavestudentsusetheseforjobs Labfee wouldbe 150 studentSimpleweb basedinterfacesexistRHadoopOnDemand HOD comingsoonInjectsnewnodesintoliveclusters SpinnakerLabs Inc MorePerformance Scalability SpinnakerLabs Inc NumberofTasks Mappers 10 nodes or3 2 cores Reducers 2 nodes or1 05 cores Twodegreesoffreedominmapperruntime Numberoftasks node andsizeofInputSplitsSeehttp wiki apache org lucene hadoop HowManyMapsAndReduces SpinnakerLabs Inc MorePerformanceTweaks Hadoopdefaultstoheapcapof200MBSet mapred child java opts Xmx512m1024MB processmayalsobeappropriateDFSblocksizeis64MBForhugefiles setdfs block size 134217728mapred reduce parallel copiesSetto15 50 moredata morecopies SpinnakerLabs Inc DeadTasks Studentjobswould runaway adminrestartneededVeryoftenstuckinhugeshuffleprocessStudentsdidnotknowaboutPartitionerclass mayhave