大数据即席查询秒级响应方案

1、大数据即席查询秒级响应方案技术创新，变革未来客户关怀和CEM市场分析360C客户洞察实时营销与推荐客户忠诚度维系客户精细分群与个性化推荐客户关怀与流程优化预测与影响力分析网络增效网络性能管理与SQM策略保障 快速决策与根因分析定位网络问题与规划数据货币化 数据变现 OTT开放竞合M2M和位置分析Router +WDM(Backbone)DDDCloudOS/OpenStack(Local Resource) + Middlewareapps for OM apps for BizAPIapps forconsumerOSS suitsBig Data SuitsBSS suitsE2E ICT

2、 Resource Orchestration EngineE2E ICT Resource Orchestration EngineRaaSNaaSOpenStack PCRFCaaSRRURRUPartnersOM TeamBizCustomerConsumerCPEMxUMxUONTSDNSmarter SoftCom 业务和运营的智能融合1SDN实时大象流挖掘5小区拥塞动态控制2IPRAN流量仿真3 SON 网络自动实时优化 4快速故障关联处理6潜在离网用户维挽 7 一站式服务优化8开放变现467OperationsBig Data Suits85Apps & ServicesClou

3、d OS/OpenStack (Local Resource, IaaS)SGSN MMEIMS HSSSDNcontrollerIT apps (SaaS)Telco apps SMS/IPTVMiddleware (PaaS)Cloud OS/OpenStack (Local Resource, IaaS)BRAS S/PGW GGSNRNCSDNSRC ControllerITappsPaaSFW DPI vCPESBC NAT1CloudEdg eCloud OS/OpenStackGSMLTEUMTS3CloudBBCloud OS/OpenStackADSLG.FastVDSL2C

4、loudDSL/OL TSmall CellSmall CellDD以太+OTN(Metro)DSDN数据现在和未来将深刻的改变运营商How to choose storage forcomplex big data requirements?NoSQL DatabaseKey-Value store: low latency, 5msCan not support multi-dimension queryMulti-dimensional problemPre-compute all aggregation combinationsComplexity: O(2n)Dimension 10

5、Too much spaceSlow loading speedShared nothing databaseParallel scan + distributed computeQuestionable scalability and fault-toleranceCluster size 100 data nodeNot suitable for big batch jobCan not integrate with Hadoop ecosystemSearch engineAll column indexedFast searchingSimple aggregationDesigned

6、 for search but not OLAPcomplex computation: TopN, join, multi-level aggregationNo SQL supportSQL on HadoopModern distributed architecture, scale well in computation.Pipeline based: Impala, Drill, Flink, BSP based: Hive, SparkSQLBUT, still using file format designed for batch jobFocus on scan onlyNo

7、 index support, not suitable for pointor small scan queriesArchitects choiceLoadingApp1App2App3ReplicationApp1App2App3Choice 1: CompromisingChoice 2: Replicating of data目录:uCarbonData项目背景和适合的场景u 关键技术介绍u 性能和DEMO演示u Apache CarbonData社区和路标客户需求：多维组合即席分析详单过滤查询按列扫描查询开源生态集成当前大数据生态系统，没有一种存储方式同时满足上面所有的需求！按列扫

8、描查询（Full Scan）：没有过滤条件，仅仅做汇 总计算等只查询几列信息典型的场景如：数据清洗处理日志分析典型场景1：按列扫描查询C1C2C3C4C5C6C7R1R2R3R4R5R6R7R8R9R10.详单过滤查询（Small Scan）：按关键字快速过滤查询 (类 似HBase)多组过滤条件组合，查询 所有列要求查询性能秒级响应典型的场景如：运维查询用户行为分析典型场景2：详单过滤查询C1C2C3C4C5C6C7R1R2R3R4R5R6R7R8R9R10即席分析/Adhoc查询：汇总计算多维度组合OLAP分析低时延即席查询典型的场景如：Dash-Board报表Ad-hoc分析典型场景3：

9、多维组合即席分析C1C2C3C4C5C6C7R1R2R3R4R5R6R7R8R9R10R11详单过滤查询(Small scan)按列扫描查询(Full scan)多维组合即席分析(OLAP analysis)CarbonData(一份数据满足所有cases)Apache CarbonData实现一份数据同时满足多种业务需求， 与Spark引擎 对接后形成一套分布式多维分析解决方案。为什么开始CarbonData项目?目录:u CarbonData项目背景和适合的场景u 关键技术介绍u 性能和DEMO演示u Apache CarbonData社区和路标v分布式能力v快速查询秒级响应v高效数据存储

10、方式v无缝与大数据生态集成开源是为了构建生态，CarbonData是数据存储层技术，要发挥价值， 需要与计算层、查询层有效集成在一起，形成E2E生态发挥最大价值。CarbonData设计思路CarbonData独特的价值特性v多种索引(MDK,MinMax,倒排)， 快速找到目标数据v字典编码，减少计算开销v支持数据更新IUD(开发中ing)v与大数据生态无缝集成，具有 HDFS分布式、可靠性等所有 优点YearsQuartersMonthsTerritoryCountryQuantitySales2003QTR1JanEMEAGermany14211,4322003QTR1JanAPACCh

11、ina54154,7022003QTR1JanEMEASpain44344,6222003QTR1FebEMEADenmark54558,8712003QTR1FebEMEAItaly67556,1812003QTR1MarAPACIndia529,7492003QTR1MarEMEAUK57051,0182003QTR1MarJapanJapan56155,2452003QTR2AprAPACAustralia52550,3982003QTR2AprEMEAGermany14411,5321,1,1,1,1 : 142,114321,1,1,3,2 : 541,547021,1,1,1,3

12、: 443,446221,1,2,1,4 : 545,588711,1,2,1,5 : 675,561811,1,3,3,6 : 52,97491,1,3,1,7 : 570,510181,1,3,2,8 : 561,552451,2,4,3,9 : 525,503981,2,4,1,1 : 144,11532多维Key索引介绍数据即索引 (multi-dimensional keys)Blocklet Logical ViewSort (MDK Index)1,1,1,1,1 : 142,114321,1,1,1,3 : 443,446221,1,1,3,2 : 541,547021,1,2

13、,1,4 : 545,588711,1,2,1,5 : 675,561811,1,3,1,7 : 570,510181,1,3,2,8 : 561,552451,1,3,3,6 : 52,97491,2,4,1,1 : 144,115321,2,4,3,9 : 525,50398Sorted MDK IndexC1 C2 C3 C4 C5C6C7111111421143211113443446221113254154702112145455887111215675561811131757051018113285615524511336529749124111441153212439525503

14、98Encoding列式索引和排序高效数据压缩(1/3)Blocklet Physical View142114324434462252550398111111111110 108103106221222429221311339333311147422111354111C1C2C3C4C5C6d rdrd rdrd rd rd rC71|1 :1|1 :1|1 :1|1 :1|1: 142:114321|2 :1|2 :1|2 :1|2 :1|9: 443:446221|3 :1|3 :1|3 :1|4 :2|3: 541:547021|4 :1|4 :2|4 :1|5 :3|2: 545:5

15、88711|5 :1|5 :2|5 :1|6 :4|4: 675:561811|6 :1|6 :3|6 :1|9 :5|5: 570:510181|7 :1|7 :3|7 :2|7 :6|8: 561:552451|8 :1|8 :3|8 :3|3 :7|6: 52:97491|9 :2|9 :4|9 :3|8 :8|7: 144:115321|10:2|10:4|10:3|10 :9|10 : 525:50398sort column within column chunk)Run Length Encoding & CompressionDim1 BlockDim2 BlockDim3 B

16、lockDim4 BlockDim5 Block8(7)9(10)Columnar Store541547021(1-10)1(1-8)1(1-3)1(1-2,4-1(1,9)545588712(9-10)2(4-5)6,9)2(3)675561813(6-8)2(7)3(2)570510184(9-10)3(3,8,10)4(4)561552455(5)5297496(8)14411532Column Level7(6)inverted Index倒排索引目录:u CarbonData项目背景和适合的场景u 关键技术介绍u 性能和DEMO演示u Apache CarbonData社区和路标测

17、试环境DEMO EnvironmentData ModelNumber of Nodes1 master + 3workers#Columns300 (150 String, 150Double)# High Cardinality ColumnsColumns (10 Million)Columns (0.5 Million)vCPU40coresMemory384G# Medium Cardinality Columns4 Columns (0.4 Million)2 Columns (0.2 Million)11 Columns (0.1 Million)Data Size1.9TB#R

18、ecords1 billion rows * 300 columns#Row Size2KB性能构造300万行数据用同样的SQL语句分别查询CSV，Parquet，CarbonData数据：benchmark csvdf.filter($name = Allen and $gender = Male and $province = NB and$singler = false).count DEMO演示: CSV , Parquet,CarbonData目录:u CarbonData项目背景和适合的场景u 关键技术介绍u 性能和DEMO演示u Apache CarbonData社区和路标Apache CarbonData社区已发布了社区稳定版本 Apache CarbonData 0.1.0,0.1.1深度解读Apache CarbonData：/cn/news/2016/07/huwei-CarbonData-data- second-resApache CarbonData源代码地址: /apache/incubator- carbondata订阅Dev Mailing，参与社区讨论：dev如果有任何需求、建议、defects反