Apache-Kylin在大数据系统中应用

ApacheKylin,OLAPonHadoop,http:/kylin.io,Agenda,WhatsApacheKylin?TechHighlightsPerformanceRoadmapQ&A,ExtremeOLAPEngineforBigData,KylinisanopensourceDistributedAnalyticsEnginefromeBaythatprovidesSQLinterfaceandmulti-dimensionalanalysis(OLAP)onHadoopsupportingextremelylargedatasets,WhatsKylin,kylin/kiln/麒麟,-n.(inChineseart)amythicalanimalofcompositeform,OpenSourcedonOct1st,2014,BeacceptedasApacheIncubatorProjectonNov25th,2014,BigDataEraMoreandmoredatabecomingavailableonHadoopLimitationsinexistingBusinessIntelligence(BI)Tools,LimitedsupportforHadoopDatasizegrowingexponentiallyHighlatencyofinteractivequeriesScale-Uparchitecture,ChallengestoadoptHadoopasinteractiveanalysissystem,MajorityofanalystgroupsareSQLsavvyNomatureSQLinterfaceonHadoopOLAPcapabilityonHadoopecosystemnotreadyyet,5,WhynotBuildanenginefromscratch?,ExtremeScaleOLAPEngine,Kylinisdesignedtoquery10+billionsofrowsonHadoop,ANSISQLInterfaceonHadoop,KylinoffersANSISQLonHadoopandsupportsmostANSISQLqueryfunctions,SeamlessIntegrationwithBITools,KylincurrentlyoffersintegrationcapabilitywithBIToolslikeTableau.,InteractiveQueryCapability,UserscaninteractwithHivetablesatsub-secondlatency,MOLAPCube,DefineadatamodelfromHivetablesandpre-buildinKylin,ScaleOutArchitecture,Queryserverclustersupportsthousandsconcurrentusersandprovidehighavailability,FeaturesHighlights,CompressionandEncodingSupportIncrementalRefreshofCubesApproximateQueryCapabilityfordistinctcount(HyperLogLog)LeverageHBaseCoprocessorforquerylatencyJobManagementandMonitoringEasyWebinterfacetomanage,build,monitorandquerycubesSecuritycapabilitytosetACLatCube/ProjectLevelSupportLDAPIntegration,FeaturesHighlights,CubeDesigner,JobManagement,QueryandVisualization,TableauIntegration,eBay,90%queryScanRange(RowKey)Aggregations-MeasureColumns(RowValues),ScanHBasetableandtranslateHBaseresultintocuberesult,HBaseResult(key+value)-CubeResult(dimensions+measures),HowtoQueryCube?,StorageEngine,Curseofdimensionality:Ndimensioncubehas2Ncuboid,FullCubevs.PartialCube,Hughdatavolume,DictionaryEncodingIncrementalBuilding,HowtoOptimizeCube?,CubeOptimization,FullCube,Pre-aggregatealldimensioncombinations“Curseofdimensionality”:Ndimensioncubehas2Ncuboid.,PartialCube,Toavoiddimensionexplosion,wedividethedimensionsintodifferentaggregationgroups,2N+M+L2N+2M+2L,Forcubewith30dimensions,ifwedividethesedimensionsinto3group,thecuboidnumberwillreducefrom1Billionto3Thousands,230210+210+210,Tradeoffbetweenonlineaggregationandofflinepre-aggregation,HowtoOptimizeCube?,FullCubevs.PartialCube,HowtoOptimizeCube?,PartialCube,DatacubehaslostofduplicateddimensionvaluesDictionarymapsdimensionvaluesintoIDsthatwillreducethememoryandstoragefootprint.DictionaryisbasedonTrie,HowtoOptimizeCube?,DictionaryEncoding,HowtoOptimizeCube?,IncrementalBuild,Streaming,ongoingeffort,Cubeisgreat,but,SometimeswewanttodrilldowntorowlevelinformationCubetakestimetobuild,howaboutreal-timeanalysis?,Streamingwithinvertedindex,streaming,Karfka,hourly/dailybatch,minutesbatch,InvertedIndexReal-timeStore,Kylin0.8,LambdaArchitectureSQLQueryHybridStorageInterface,CubeHistoricStore,http:/kylin.io,Agenda,WhatsApacheKylin?TechHighlightsPerformanceRoadmapQ&A,Kylinvs.Hive,#,Query,Type,ReturnDataset,Query,OnKylin(s),Query,OnHive(s),Comments,1,HighLevel,Aggregation,4,0.129,157.437,1,217times,23,AnalysisQueryDrillDownto,Detail,22,669325,029,1.61512.058,109.206113.123,68times9times,4,DrillDownto,Detail,524,780,22.42,6383.21,278times,5,DataDump,972,002,49.054,N/A,100500,200150,SQL#1,SQL#2,SQL#3,HiveKylin,HighLevelAggregation,AnalysisQuery,DrillDowntoDetail,LowLevelAggregation,TransactionLevel,Basedon12+Brecordscase,Performance-Concurrency,Linearscaleoutwithmorenodes,Performance-QueryLatency,90%queries5s,GreenLine:90%tilequeriesGrayLine:95%tilequeries,http:/kylin.io,Agenda,WhatsApacheKylin?TechHighlightsPerformanceRoadmapQ&A,KylinEvolutionRoadmap,2015,2014,2013,Initial,PrototypeforMOLAP,BasicendtoendPOC,MOLAPIncrementalRefreshANSISQLODBCDriverWebGUIACLOpenSource,HOLAP,NextGenLambdaArchAutomation,StreamingOLAPJDBCDriverNewGUIExcelSupportmore,CapacityManagementIn-MemoryAnalysis(TBD)Spark(TBD)more,TBD,Future,Sep,2013,Jan,2014,Sep,2014,H1,2015,KylinCore,FundamentalframeworkofKylinOLAPEngineExtensionPluginstosupportforadditionalfunctionsandfeaturesIntegrationLifecycleManagementSupporttointegratewithotherapplications,InterfaceAllowsforthirdpartyuserstobuild