Serengeti-虚拟化你的大数据应用41.ppt

2009VMwareInc Allrightsreserved Serengeti 虚拟化你的大数据应用 蔺永华Vmware Inc Agenda Today sbigdatasystem Whyvirtualizehadoop Serengetiintroduction Commonquestionsaboutvirtualization Serengetisolution DeepinsightintoSerengeti Summary Q A Today sBigDataSystem ETL UnstructuredData HDFS RealTimeStructuredDatabase BigSQL Data ParallelBatchProcessing RealTimeStreamsReal TimeProcessing s4 storm Analytics Agenda Today sbigdatasystem Whyvirtualizehadoop Serengetiintroduction Commonquestionsaboutvirtualization Serengetisolution DeepinsightintoSerengeti Summary Q A ChallengesToUseHadoopinphysicalinfrastructure Deployment Difficulttodeploy costseveralpeopleforseveraldaysevenmonths Difficulttotuneclusterperformance LowEfficiency Hadoopclustersaretypicallynot100 utilizedacrossallhardwareresources Difficulttoshareresourcessafelybetweendifferentworkload SinglePointofFailure SinglepointoffailureforNameNodeandJobtracker NoHAforHive HCatalog etc WhyVirtualizeHadoop GetyourHadoopclusterinminutes 1 1000humanefforts LeastHadoopoperationknowledgeFullyautomatedprocess 10minutestogetaHadoop HBaseclusterfromscratch ServerpreparationOSinstallationAutomatebySerengetionvSpherewithbestpracticeNetworkConfigurationHadoopInstallationandConfigurationManualprocess costdays WhyVirtualizeHadoop Consolidatesprawlingclusters Clustersshareserverswithstrongisolation SingleHardwareInfrastructure Unifiedoperations Optimize SharedResources higherutilization Elasticresources fasteron demandaccess HadoopDev HadoopProd HBase ClusterSprawlingSinglepurposeclustersforvariousbusinessapplicationsleadtoclustersprawl ClusterConsolidation Simplify Finance Hadoop VirtualizationPlatform HadoopDev HadoopProd HBase PortalHadoop PortalHadoop 30 CAPEXDown 50 resourcesaresittingidlewhilehighpriorityjobisburningupitscluster Utilizeallresourcesfrompoolondemand Dynamicelasticscalingonsharedresourcepool WhyVirtualizeHadoop Utilizeallyourresourcestosolvethepriorityproblem3Xfastertogetanalyticresults vSphereHighAvailability HA protectionagainstunplanneddowntime Overview ProtectionagainsthostandVMfailures Automaticfailuredetection host guestOS Automaticvirtualmachinerestartinminutes onanyavailablehostincluster OSandapplication independent doesnotrequirecomplexconfigurationchanges Coordination Zookeepr ManagementServer HighAvailabilityfortheHadoopStack HadoopDistributedFileSystem HBase Key Valuestore HDFS MapReduce JobScheduling ExecutionSystem Pig Data Flow Hive BIReporting ETLTools RDBMS JobtrackerNamenode SQL HiveMetaDB HCatalogHcatalogMDB Server XX HAHA AppOS AppAppOSOS AppOS AppOS AppOS AppOS VMwareESXX VMwareESX Zerodowntime zerodatalossfailoverforallvirtualmachinesincaseofhardwarefailures IntegratedwithVMwareHA DRS Nocomplexclusteringorspecializedhardwarerequired Singlecommonmechanismforallapplicationsandoperating FT vSphereFaultToleranceprovidescontinuousprotectionOverview SingleidenticalVMsrunninginlocksteponseparatehosts systemsZerodowntimeforNameNode JobTrackerandothercomponentsinHadoopclusters Agenda Today sbigdatasystem Whyvirtualizehadoop Serengetiintroduction Commonquestionsaboutvirtualization Serengetisolution DeepinsightintoSerengeti Summary Q A EasyandrapiddeploymentandmanagementOpensourceprojectlaunchedinJune2012 0 8isreleasedatApr andwillrelease0 9atJun ToolkitthatleveragevirtualizationtosimplifyHadoopdeploymentandoperationsDeployaclusterin10MinutesfullyautomatedCustomizeHadoopandHBaseclusterAutomatedclusteroperation Comewitheco systemcomponentsSupportallpopularHadoopDistributions Serengeti Demo 10minutestoaHadoopclusterwithSerengeti Agenda Today sbigdatasystem Whyvirtualizehadoop Serengetiintroduction Commonquestionsaboutvirtualization Serengetisolution DeepinsightintoSerengeti Summary Q A CommonquestionsaboutvirtualizationLocalDisk Canlocaldiskbeusedinvirtualizationenvironment FlexibilityandScalabilityHowtoflexiblescheduleresourcesbetweenclustersanddifferentapplicationsasmentionedabove DatastabilityInvirtualenvironment howcanwedistributedataacrosshostandrack DatalocalityHadoopwillschedulecomputetasksnearbythedata toreducenetworkIOfordataR W Canvirtualenvironmentgetthesameresult PerformanceHowabouttheperformanceinvirtualenvironment Agenda Today sbigdatasystem Whyvirtualizehadoop Serengetiintroduction Commonquestionsaboutvirtualization Serengetisolution DeepinsightintoSerengeti Summary Q A CanIuselocaldiskeasily OtherVM OtherVM OtherVM OtherVM OtherVM OtherVM OtherVM OtherVM Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop SerengetiExtendVirtualStorageArchitecturetoIncludeLocalDisk SharedStorage SANorNAS Easytoprovision Automatedclusterrebalancing HybridStorage SANforbootimages otherworkloads LocaldiskforHadoop HDFS Host Host Host Host Host Host Howtoflexiblescalein scaleout Howtoflexiblescheduleresourcesbetweenclustersanddifferentapplications Compute CurrentHadoop T1 T2 VM VM VM VM CombinedStorage ComputeHadoopinVM VMlifecycledeterminedbyDatanode Limitedelasticity VMStorageSeparateStorage VMStorageSeparateComputeClusters Separatecompute fromdata Removeelasticconstrain byDatanode Elasticcompute Raiseutilization Separatevirtualcompute Computeclusterpertenant StrongerVM gradesecurityandresourceisolation EvolutionofHadooponVMs Data ComputeseparationSlaveNode SerengetiNodeScaleOut ScaleIn NameNodeHost DHost JobTracker CC CC DHost CC CC DHost CC CC DHost CC CC SerengetiBallooningEnhancementforJavaApplication JVM GuestOS Host JVM GuestOS Host GuestOSJVM Howtokeepdatastability HowtoaccessdatalocallyifdatanodeandcomputenodearelocatedindifferentVM Datanodeandtasktrackercombinedcluster DataComputeseparatedcluster masterHost workerHost workerHost masterHost DatanodeHost Tasktracker DatanodeHost Tasktracker TasktrackerTasktrackerDatanodeHost Computeonlycluster1Computeonlycluster2HDFSclusterComputeOnlycluster Rack1 Rack2 Rack1 DistributedandData ComputeAssociatedVMPlacementRack2 Rack1JobtrackerJobtrackerNamenodeHost Rack2TasktrackerTasktrackerDatanodeHost HadoopTopologyChanges forVirtualization HadoopTopologyAwareness SerengetiHVE D1 D2 R1 R2 N1 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 R3 R4 3 D1 D2 R1 R2 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 R3 R4 2 3 N2 N3 N4 N5 N6 N7 N8 11 23 2 11 2 3 4 HADOOP 8468 UmbrellaJIRA HADOOP 8469HDFS 3495HDFS 3498 Hadoop NetworkTopologyExtension HadoopVirtualizationExtensionsforTopologyHVETaskSchedulingPolicyExtensionBalancerPolicyExtensionReplicaChoosingPolicyExtensionReplicaPlacementPolicyExtensionReplicaRemovalPolicyExtension HDFS MapReduce HadoopCommon MAPREDUCE 4310MAPREDUCE 4309HADOOP 8470HADOOP 8472 Istheresignificantperformancedegradationinvirtualizationenvironment Isthereanyperformancedata VirtualizedHadoopPerformance NativeversusVirtualPlatforms 32hosts 16disks host Source Agenda Today sbigdatasystem Whyvirtualizehadoop Serengetiintroduction Commonquestionsaboutvirtualization Serengetisolution DeepinsightintoSerengeti Summary Q A UIClientFlexUI SerengetiarchitecturediagramCLIClientSpringShellSerengetiWebService Hibernate DAO vPostgres VCadapter IronfanserviceThriftServiceProgressIronfanreportChefserver RestAPI Cookbook VHMstep RabbitMQ VMruntimeManager Host Host Host Host Host VirtualizationPlatform HadoopNode ChefClientHAkit HadoopNode HadoopNode Package repository vCenter CustomizingyourHadoop HBaseclusterwithSerengeti Choiceofdistros Storageconfiguration ChoiceofsharedstorageorLocaldisk Resourceconfiguration Highavailabilityoption ofnodes distro apache groups name master roles hadoop namenode hadoop jobtracker storage type SHARED sizeGB 20 instance type MEDIUM instance num 1 ha true name worker roles hadoop datanode hadoop tasktracker instance type SMALL instance num 5 ha false OnecommandtoscaleoutyourclusterwithSerengeti clusterresize name nodegroupworker instanceNum Configure reconfigureHadoopwitheasebySerengeti ModifyHadoopclusterconfigurationfromSerengeti Usethe configuration sectionofthejsonspecfile SpecifyHadoopattributesincore site xml hdfs site xml mapred site xml hadoop env sh log4j properties ApplynewHadoopconfigurationusingtheeditedspecfile configuration hadoop core site xml checkforallsettingsathttp hadoop apache org common docs r1 0 0 core default html hdfs site xml checkforallsettingsathttp hadoop apache org common docs r1 0 0 hdfs default html mapred site xml checkforallsettingsathttp hadoop apache org common docs r1 0 0 mapred default html io sort mb 300 hadoop env sh HADOOP HEAPSIZE HADOOP NAMENODE OPTS HADOOP DATANODE OPTS clusterconfig namemyHadoop specFile home serengeti myHadoop json FreedomofChoiceandOpenSource CommunityProjects Distributions Flexibilitytochoosefrommajordistributionsclustercreate namemyHadoop distroapache Supportformultipleprojects Openarchitecturetowelcomeindustryparticipation ContributingHadoopVirtualizationExtensions HVE toopensourcecommunity HDFS2withNamenodeFederationandHADeployCDH4Hadoopcluster NameNodeFederation NameNodeHA MapReducev1 HBase Pig Hive andHiveServer CDH4configurations ScaleoutElasticity JobTrackerHA FT ActiveNamenode StandbyNamenode ActiveNamenode StandbyNamenode ZookeeperGroup ZK ZK ZK CoordinateNamenodeGroup1 CoordinateNamenodeGroup2 Quorum basedmetadatastore DataNodesDat