Jstorm开源技术架构最佳实践

1、Jstorm开源技术架构最佳实践Large-Scale Stream Processing in Alibaba目录发展历史使用场景架构JStorm增强性能稳定性新特性开源社区JStorm Future发展历史2013/4/300.7.1Java core Scheduler 1.02013/12/300.9.0Nimbus HA 1.0Backpressure 1.0Scheduler 2.02015/11/122.1.0Donate to apache Support Storm 0.10 binaryMetrics Redesign Koala 3.02015/10/270.9.8.1K

2、oala 2.0Scheduler 4.0Backpressure 2.0 Topology Master2014/10/110.9.6.3Koala 1.0Scheduler 3.0 User Defined LogUser Defined Metrics 1.0JStorm Milestones从2013-04-30 起发布了超过25个版本0.7.1/0.9.0/0.9.1/0.9.2/0.9.3/0.9.3.1/0.9.4/0.9.4.1/0.9.5/0.9.5.1/0.9.6/0.9.6.1/0.9.6.2/0.9.6.3/0.9.7/0.9.7.1/0.9.7.2/0.9.8/0.9

3、.8.1/0.9.8.2/2.0.4/2.1.0/2.1.1/2.2.0/2.2.1/2.3.02016/5/312.2.0Improve performance Exactly Once Sliding WindowShade Dependency Backpressure 3.0使用场景JStorm在阿里现状阿里集团、蚂蚁、优土、高德4000+机器StandaloneJStorm-on-YARNJStorm-on-docker1500+应用，2000+拓扑2PB+数据/天应用场景反欺诈：Nut/Velocity审计：阿里妈妈广告/P4P，AMG数据统计分析：Eagleeye，AE，BI监控

4、：Tlog，Rds-monitor，Oceanbase-monitor, Cainiao Radar, YunOSMonitor数据同步：RDS-Log-Sync, Unify-Log实时推荐：Alipay 1315, Alipay Hyperloop, TPP应用调度：AE邮件实时分析，网销宝架构JStorm架构TLogKeplerNUTARMRDSTripYunOSCainiaoPortal (Koala)SQL enginePlugin（Kafka/Metaq/TT/Mysql/HDFS/Hbase/）CEP PluginTridentExactly OnceExtension APIR

5、aw Storm APIJStorm CoreLocalCluster Standalone, YARN, FuxiCloud (Docker, ECS, .)JStorm增强性能4206108122 worker, 1 spout, 2 split, 2 count10 worker, 5 spout, 20 10 worker, 10 spout, 40 20 worker, 10 spout, 80 split, 20 countsplit, 40 countsplit, 80 count40 worker, 10 spout, 80 worker，10 spout，160 split,

6、 160 count320 split，320 countMillionJStorm StormHW: 32 CPU core/128GTest: Storm FastWordCountStorm version: 1.0.2JStorm version: 2.2.0Worker: 6G RAMHow do we optimize?PipelineApplica tionDeployCoreTask Pipeline TuningUserDefined schedulerImprove efficiencyImprove parallelShuffleResolve bottleneckBat

7、ch DesignSerialization TuningSeparate spoutsend/recvMetrics TuningRouting OptimizationOverheadReduction稳定性隔离HA监控升级/降级多集群部署反压热部署/灰度发布Tuple lifecycle自定义metrics监控报警(Koala)单点故障: nimbus HA节点自检: checkhealth超大规模拓扑: topology-master跨集群: jstorm-on-docker/standalone单集群: jstorm-on-yarn/cgroup异地: 多链路同城多机房: zk监控系

8、统自定义监控大规模部署NimbusTopology MasterTaskZKClusterTopologyTaskTaskTopology MasterTaskTaskTaskTopology MasterTaskTopologyTaskTaskTask heartbeatControl eventTopologyTopology MasterslaveTopology MasterMetrics反压Netty clientNetty cacheSending taskNetty serverTask cacheReceive taskIn one workerTask cacheReceiv

9、e taskExceed Threshold灰度发布&热升级Exactly-OnceChandy-Lamport AlgorithmBarrierStream AligningSpoutStateless boltState boltEnding boltExactly-OnceTopology MasterHdfs/hbaseControllerNext tuple CommitExecuteExecuteCommit Cache dbExecuteExecute Cache dbCommit offsetNormal batchCheckpoint barrierCheckpoint ba

10、rrierJStorm checkpoint (optional)Jstorm checkpointCheckpoint barrierCheckpoint barrierCheckpoint and clear old checkpointSpoutStateless boltState boltEnding boltExactly-OnceTopology MasterHdfs/hbaseControllerNext tuple RollbackExecuteExecuteRollback Cache dbExecuteExecute Cache dbRollback barrierRollback barrierRollback barrierRollback barrierRead last success checkpoint and clean useless checkpoint其他feature自定义调度动态调整日志级别自定义日志用户自定义metrics优雅shutdown动态扩容、更新配置新的窗口机制重构的窗口机制，支持