高级模板ppt课件.pptx

CortanaIntelligenceSuiteHDInsight ComprehensivesetofmanagedApachebigdataprojects CoreEngine BatchMapReduce ScriptPig SQLHive NoSQLHBase StreamingStorm In MemorySpark MachineLearningandAnalytics HDInsight HadoopandSpark StreamAnalytics DataLakeAnalytics MachineLearning AzureAnalysisServices HDInsight Whatisit 3 AstandardApacheHadoopdistributionofferedasamanagedserviceonMicrosoftAzure BasedontheHortonworksDataPlatform HDP ProvisionedasclustersonAzure ClusterscanrunonWindowsorLinuxServers Offersacapacity on demand pay as you gopricingmodelIntegrateswith AzureBlobStorageandAzureDataLakeStorefortheHadoop HDFS AzurePortalformanagementandadministrationVisualStudioforapplicationdevelopmenttooling Inadditiontothecore HDInsightsupportstheHadoopEcosystem HDInsight How Whereitfits 4 analyticsservice U SQL YARN HDFS Store IntegratedanalyticsandstorageFullyManagedEasytouse dialforscale ProvenatscaleAnalyzedataofanysize shapeorspeedOpen standardsbased Partners Hortonworks Cloudera MapR Analytics clusters HDInsight AzureDataLake AzureDataLakeStore HDInsightclusterarchitecture AzureVNet SecuregatewayAuthNHTTPProxy HighlyavailableHeadnodes x Workernodes BenefitsofHadoopasaAzureService 6 HDInsightclusterscanbeconfiguredinjustafewminutesThroughtheHDFSinterface HDInsightcanoperatedirectlyondatastoredinAzureBlobStorageorAzureDataLakeStore AseparateanddedicatedHDFSclusterisnotrequired Benefitsinclude DatareuseandsharingElasticscale outLowerdatastoragecostsProtectionagainstdatalossHighPerformance Thehigh speedflatnetworksinAzuredatacentersprovidefastaccessbetweenthevirtualmachinesintheclusterandAzureBlobStorageorAzureDataLakeStoresodatamovementisveryefficientCommissionanddecommissionHDInsightclustersatwillVisualizethedatastoredinHivetablesusingExcelandthesetofadd ins suchasPowerQuery PowerView PowerPivot andPowerMap AgendaHCatalogEssentialsHcatalogComponents HCatalog 7 Hcatalog Whatisit 8 HCatalogdisplaysdatafromRC textfiles orsequencefilesinatabularview IsatablemanagementlayerthatexposesHivemetadata asRESTAPIs tootherHadoopapplications HCatalog stableabstractionpresentsuserswitharelationalviewofdatainHDFSEnsuresusersneednotworryaboutwhereorinwhatformattheirdataisstored Benefits Freestheuserfromhavingtoknowwherethedataisstored withthetableabstractionEnablesnotificationsofdataavailabilityProvidesvisibilityfordatacleaningandarchivingtools IsametadataandtablemanagementsystemfortheHadoopplatformthatenablessharing HCatalog Howitworks 9 HCatalogsupportsreadingandwritingfilesinanyformatforwhichaHiveSerDe serializer deserializer canbewrittenTouseacustomformat youmustprovidetheInputFormat OutputFormat andSerDe HCatalogisbuiltontopoftheHivemetastoreandincorporatescomponentsfromtheHiveDDL TheHCataloginterfaceforPigconsistsofHCatLoaderandHCatStorer whichimplementthePigloadandstoreinterfacesrespectively TheHCataloginterfaceforMapReduce HCatInputFormatandHCatOutputFormat isanimplementationofHadoopInputFormatandOutputFormat HcatalogDataModelandhierarchyisthesimilartotheHBasemodel Diagramsource Yahoo PresentationatHadoopSummit2014 AgendaHiveEssentialsHiveArchitectureHiveQLHiveVsRDBMS Hive 10 ApacheHIVE Whatisit 11 Enablesyoutoviewbigdataas Tables andusefamiliarSQL likeoperationsonthem Letsyouimposeatable likeviewonbigdataHowever notasubstituteforfull fledgedRDBMS HivetablescanbequeriedusingHiveQL aSQL likequerylanguageHiveQLqueriesareeventuallyexecutedasMapReduceorTezprogramsbyHadoop A DataWarehouse builtusingHadoopforqueryingandanalyzingbigdata Hive QueryExecutionComponents 12 HiveQL ExecutionFlow 13 Hadoop HDFS Driver Execution 1 2 5 3 5 1 HiveclientssendHiveQLqueriestothedriverThedriversendsquerytothecompilerforparsingandsyntaxchecking Thecompilergetsthemetadatafromthestoreandreturnsaexecutionplantothedriver Thedriversendstheexecutionplantotheexecutionengine TheexecutionenginesubmitstheplanasaMRjobtobeexecutedbyHadoop AfterHadoopcompletesexecution theexecutionenginegetstheresultswhicharereturnedtothedriver whichinturnreturnstheresultstotheHiveclients TheexecutionenginecanexecutemetadataoperationsagainsttheMetastore 4 1 2 5 3 5 1 4 HiveDataOrganizationHierarchy 14 SampleData CustomersFile 15 CustomerId CustomerName EmailPreferences BillingAddress ShippingAddress SampleData OrdersFile 16 CustomerId OrderDataandTime OrderId ProductDetails HiveTableCreation 17 Hivesupportscomplextypessuchas map HDInsightHiveTablescandirectlypointtoAzureDataLakeStorefiles Ordertablecreationisshowninnotessections LoadingDataintoHiveTables 18 Hadoop HDFS hive warehouse customer Options Loadingdataintoorderstableisshowninthenotessections CustomersTable 19 HiveQLOverview 20 HiveQL SQL likesyntax SELECT FROM WHEREALLandDISTINCTSupportforPartitioningGROUPBYSORTBY ORDERBY CLUSTERBY DISTRIBUTEBYJOINs LEFT RIGHT FULL OUTER LEFTSEMI CROSSJOINBuilt in Operators Functions AggregateFunctionsUserDefinedFunctions UDFs SupportforSubqueriesWindowingandAnalyticsFunctions HiveQL 21 Query Findnameandbillingcityofcustomerswithshippingaddressesin TX or IL GroupbyCity HiveQL 22 Query FindnamesofproductspurchasedbycustomerswhoseshippingaddressisinTexas HDIonLinux HiveQLExecutionChoices 23 ComparingRDBMSandHive 24 RecentHivereleases Hive0 10 BATCH Hive0 13 INTERACTIVE Hive0 14 1 0 SUB SECOND Read onlyData HiveQL MR Read onlyData SubstantialSQL MR Tez Modifyw Transactions MR Tez Spark SQL 2011Analytics EnterpriseSQLatHadoopScale Stinger 25 Hive1 2 SQLEnhancements OptimizerImprovements Security ImprovedPerformance VisualStudiotooling ShipswiththeAzureSDKSupportedforVS2012 2013and2015Ena