湖仓一体全面开启实时化时代

DatawarehousesApplicationsDatabaseETLDatawarehousesApplicationsDatabaseETLDataExploreETLDatawarehousesstructured,semi-structuredandunstructuredDataRealtimeAnalyticsMachineDatasciencestructuredstructured,semi-structuredandunstructuredDataAnalyticsMachineLearningDatascienceAlakeformatthatenablesbuildingaRealtimeLakehouseArchitecturewithFlinkandSparkforbothstreamingandbatchoperationsStreamingIngestionstreamingstreamingstreaming001011101010101010101010101001001回阿里巴巴集团数据业务回阿里巴巴集团数据业务RealtimeComputeMaxComputeHologresE-MapReduce001011011010101010101010101010010ThanksApachePaimon统⼀⼤数据湖存储底座李劲松/ApachePaimonPMCChair010102020303040401 Warehouse:天级 StreamingLakehouse:分钟级 Warehouse:天级 StreamingLakehouse:分钟级 LakehouseLakehouse:天级/小时级实时数据湖02流式数据湖03国人主导社区03000250020005000250020005000构建流式链路构建流式链路02一套架构，三种计算StreamingIngestionstreamingstreamingstreamingBatchBatchBatch001011101010101010101010101001001流流批批03拥有全部能力的是Flink，是star04454545进展进展ServerlessServerless·所有计算引擎打通paimon达成state-of-Art的性能·结合Flink构建流批一体的streamingLakehouse·统一存储，按照不同需求使用不同的计算引擎MaxComputeSparkServerlessMaxComputeSparkServerlessFlinkServerlessStarRocks010101010101011010101010010101001011101·查询脚本：开发过程查询数据及系到最小5秒内返回·作业运维：支持启动点位的设置；面向stopwithsavepoint批调度：支持基本的批工作流与调度，以支撑简单的数据回刷查询查询...);dw.order_dw.dwd_ordersSELECT……查询性能：内置Jar&向量化读取写入能力：完整支持写入及覆盖写入Thanks010102020303TransactionsAlgorithmdata…DatabaseFileSystem…OfflineOfflineSystemStreamProcessingBatchProcessingFileSystem…SearchEngineAdvertisingEngineRecommendationEngineSampleEngine…应用规模增量延迟应用规模增量延迟支持双11作业规模 ……解决Lambda架构缺点 —//--………PaimonTableTable1Table2Cache消息队列FlinkBatchJobHTTP…PaimonTableTable1Table2Cache消息队列FlinkBatchJobHTTP 作为Cache使用TableTablen…所以分了十多张KV表存图片Cache，用户使用不便。二、新链路预期收益所以不存在Build链路不稳定的情况。…………AppnApp1…AppAppnApp1…AppnApp2…AppApp2PaimonTablePaimonTableScanSnapshot-1ScanSnapshot-1Snapshot-3Snapshot-3Snapshot-2AppApp11、并发有上限限制，吞吐受限，而盲目加并发有拉挂库的风险。2、核心库拉取时间只能晚上。二、新链路预期收益ScanSnapshot-nScanSnapshot-1…ScanSnapshot-nScanSnapshot-1……/confluence/display/PAIMO……………FilesFilesFilesTableschainFilesrebalancePickCloneTables…FilesFilesFilesTableschainFilesrebalancePickCloneTables（database.table）Database.Table…Recreate…RecreateSnapshotRecreateRecreateSnapshotRecreateRecreateSnapshotparallelism=nThanks010102020303…01蚂蚁Paimon应用场景u简化研发模式,降低研发成本u节约了80%的存储资源以及70%的计算资源u在最后⼀批发奖的过程中，节约了1.2⼩时的核对时间，提⾼了