超大规模实时数仓架构挑战与实践解析课件

超大规模实时数仓架构挑战与实践解析技术创新

变革未来超大规模实时数仓架构挑战与实践解析技术创新变革未来1数据库发展RDBMS[SQL+OLTP

]Data

warehouseData

Cube[ETL+OLAP

]GraphVectorTextStructured

DataHeterogeneous

DataRDBMSNoSQL/NewSQLDB[Multi-Model+HTAP

]Structured

DataStructuredDataTime

SeriesSpatialData数据库发展RDBMSDatawarehouseData2阿里巴巴

－

数据库阿里巴巴－数据库3OLAP-

OLTPOn-LineTransaction

ProcessingShort-lived

tr(ns()tions(Typi)(llySm(lld(t(())ess

footprint(OftentimeRepetitive

oper(tionsOn-Linenal-tical

ProcessingLon-runnin-

queriesComplexjoin

oper(tionsExplor(toryqueriesth(tm(y())ess(l(r-e(mountofd(t(OLAP-OLTPOn-LineTransaction4数据库发展

－

架构数据库发展－架构5数据库发展

－

核心技术组件数据库发展－核心技术组件6数据库发展

－

更高的应用诉求数据库发展－更高的应用诉求7数据库发展—技术挑战数据库发展—技术挑战8整体架构支持任意条件组合的高并发低延时查询支持大吞吐数据写入兼容MySQL生态支持高可用（在线升级/扩缩容，单点故障应用透明）整体架构支持任意条件组合的高并发低延时查询9执行调度WorkloadPerQueryFairSchedulerPerTask

WeightScheduler场景混合负载10-20XPerformanceImprove执行调度Workload场景混合负载10-20XPerfor10内存管理内存调度：统一内存结构，定长块内存池化,

Binary

Process，弹性分配，支持落盘内存管理内存调度：统一内存结构，定长块内存池化,Binar11自研优化器自研优化器12GPU引擎JIT代码生成和IR层优化支持CPU和GPU运行时加速库Snappy解压高速排序高并发显存管理CPU-GPU堆外内存传输SSD至显存直传Compute

NodeGPU引擎JIT代码生成和IR层优化ComputeNode13GPU引擎

－

性能GPU引擎－性能14SQL

OptimizationMemory

optParameterizationMonitoringTuningAuto-adminDiagnosisRestoreAnomaly

DetectionSecure

proxyProtectionPatchIdentificationThreat

DetectionAlertElasticitySchedulingResource

predictionBackup

RestoreOperationsMonitoringHA、DisasterRecoveryUpgradeExpansionandcontractionWholestage

OptResourcemanagementDatabase

Self-DrivingMetadataTaskmanageDatacollectionSecureReliableCost-effectiveEfficientML

Algorithms自动化运维SQLOptimizationMemoryoptPara15新硬件RDMANVM3DX

PointOpen-Channel

SSDAPPSFile

systemFT

LOpenChannel10

LibraryNVMe

DriverOpenChannelFirmwareSSD

ControllerGPU/FPGA新硬件RDMANVMOpen-ChannelSSDAPPS16Forrester-

ContendersGartner–Niche

Player业界认可Forrester-ContendersGartner17标准测试

-

TPC-DSRealistictable

scalingRealworlddata

contentNon-uniform

distributionsComplexrelationships(Factto

Fact)标准测试-TPC-DSRealistictables18TPC-DS

R&port &

VLDBVLDB2019Analyt