构建可扩展可演进大数据架构-毕洪宇

构建可扩展可演进大数据架构,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC毕洪宇,Agenda,背景介绍及目标,数据平台,数据应用,团队,总结,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,关于饿了么,16亿元+,08年,10年,12年,14年,16年,交易2,年份,800,600,400,200,城+数,年份,700+,.均交易2,10城+,008年,10年,12年,14年,16年,日均交易116亿元+,日0单+500万+,覆.城市700+,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,background,越来越多的场景需要数据驱动决策,人难招,业务快速发展GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC数据需求越来越强,motivation,可扩展，为持续演进提供基础,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC可演进，形成闭环,数据平台,TechArchitecture:,数据服务,数据引擎,数据基础架构,Persistenttfm*,DataInfrastructure,adhoct!m*DataService,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCDataEngine,查询平台,运行状态大屏,保护机制,笛卡尔积，分区检查,资源检查,splitsizetuning,多数据源，查询引擎切换GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,数据视图引擎,SQL即接口,SQL即报GITC表GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,报表类型,细粒度权限,细粒度开关,多控件图表支持GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC可拖拽式开发,SQL生成图表,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,drag&drop方式,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,查询引擎选型,MMR与MPP的区别,分析型,交互型,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,OLAPengine:kylin,商业常品对接方便,基于查询热度构建Cube,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,大数据量存储引擎选型,MySQLDAL：核心调用路径,Cassandra：非核心调用路径,Redis-Cluster：高速k-vstore,Elasticsearch：多维度聚合查询,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,调度引擎,并行抽取，支持分库分表,支持多种数据库推送：MySQL/HBase/Cassandra/Redis,将guideline融入到工具中：自动依赖,关键路径性能分析,所有任务情况可回溯,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,数据应用,算法应用,实时业务GITC看GITCG板ITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,实时业务看板,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,计算链路,解析复用,HA,MySQL,Nginx,BinlogParser,Flume,Ka8a,Storm,MySQL,RedisCluster,DataPipeline,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,扩展存储：,手动分片三层架构自动分片,本地化聚合,多流异步合并,实时计算,Storm可扩展优化,实时存储,Redis-luster,实时监控,集群性能监控,容量-析,水位系统,扩展计算：GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,计算领域CAP,成本,高吞吐低时延,精准度,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,算法应用,个性化美食推荐,红包营销,物流调度,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,实验平台,驱动算法快速迭代,算法验平台,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,abkey-point,流量分配,流量shuffle,样本量,异常监控,multi-layer请求复用GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCmulti-layer使用正交的hash函数,团队工具,jira,workflow：bug管理，敏捷看板，变更管理，任务管理,report工具：资源图，to-dolist，变更日报等；,wiki：知识共享，SOP，故障报告管理；GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC,softskill,沟通：会议必须可落地&结果确认需落地&信息分类,failforward:DERP,SceneProtection,Escala%on,Remedia%on,Preven%on,GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITDCeGItTeCcGI%TCoGnITCGITCletsdatatalk:e.g.alertreport,总结,nomeasure，noimprovement,automateeverything,everythingmayfail,conwayslaw,balancedesign，thenoptimisationGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGIT