OushuDB技术架构演进之路

1、OushuDB技术架构演进之路CONTENTS目 录1OushuDB简介PostgreSQL传统MPP数据库HAWQ1.0HAWQ2.0OushuDB3.0OushuDB4.0OushuDB简介2011原型系 统Apache顶级项目HAWQ企业版本原生Hadoop并行SQL引擎面向企业的分析型数据库 拥有比传统MPP数据库更先进的弹性引擎 全 新 执 行 器 引 擎 Hornet 原生分布式存储MAGMA2013HAWQ1.02015HAWQ2.0Apache HAWQ孵化2018正式毕业 成为顶级 项目OushuDBPostgreSQLServerpostmasterbackendShare

2、d BuffersWALBuffersWork MemMemory OtherBuffersWriterWALWriterArchiverData FilesWALFilesLog FilesArchive FilesOther Process单机版关系型数据库PG9.6以后查询内部可以并行Master HostMaster InstanceGlobal CatalogTransaction LogsMPPStandby HostStandby InstanceGlobal CatalogTransaction LogsSegment Host1Segment 0 (Primary)Local

3、 System CatalogLocal Transaction LogsSegment 1 (Mirror)Local Table DataSegment Host2Segment 1 (Primary)Local System CatalogLocal Transaction LogsSegment 2 (Mirror)Local Table DataSegment Host3Segment 2 (Primary)Local System CatalogLocal Transaction LogsSegment 0 (Mirror)Local Table DataGreenplum Dat

4、abase架构图(2003年)完全share-nothing的架构高可用通过primary-mirror实现Segment有状态两阶段提交实现分布式事务计算存储紧耦合HAWQ1.0Master HostMaster InstanceGlobal CatalogTransaction LogsStandby HostStandby InstanceGlobal CatalogTransaction LogsSegment Host1Segment 0Local read-only System CatalogAO表数据放到HDFS上通过native c+ lib访问HDFSHDFS保证数据高可用

5、禁掉Non-AO表移除segment上本地事务日志泳道模式实现HAWQ事务实现HDFS Truncate功能移除segment上本地系统表Metadata Dispatch无状态segment计算存储物理上解耦合Segment 1Segment Host2Segment 2Local read-only System CatalogSegment 3Segment Host3Segment 4Local read-only System CatalogSegment 5HDFSHAWQ1.0架构图seg 0seg 1seg 2seg 3seg 4seg 5HAWQ2.0Master HostM

6、aster InstanceGlobal CatalogTransaction LogsStandby HostStandby InstanceGlobal CatalogTransaction LogsSegment Host1Local read-only System Catalog每张表在HDFS上一个目录虚拟segments全新Dispatcher模块块级别文件存储HDFS元数据缓存全新的资源管理模块全新的容错服务全新的管理工具动态加减节点计算存储逻辑上解耦合HDFSHAWQ2.0架构图PhysicalSegmentvsegvsegvsegSegment Host2Local rea

7、d-only System CatalogPhysicalSegmentvsegvsegvsegSegment Host3Local read-only System CatalogPhysicalSegmentvsegvsegvsegblockblockblockblockOushuDB3.0Master HostMaster InstanceGlobal CatalogTransaction LogsStandby HostStandby InstanceGlobal CatalogTransaction LogsSegment Host1Local read-only System Ca

8、talog全新Hornet执行引擎： SIMD/New hardware，性能 提升10倍可插拔存储：很容易添加外 部数据源，并且可以高效访 问External System(through Pluggable Storage)OushuDB3.0架构图PhysicalSegmentvsegvsegvsegSegment Host2Local read-only System CatalogPhysicalSegmentvsegvsegvsegSegment Host3Local read-only System CatalogPhysicalSegmentvsegvsegvsegOushuD

9、B4.0Master HostMaster InstanceGlobal CatalogTransaction LogsStandby HostStandby InstanceGlobal CatalogTransaction LogsSegment Host1Local read-only System CatalogOushuDB4.0架构图Magma作为OushuDB原生存 储，和HDFS并驾齐驱分布式存储模块Magma和新 执行器模块Hornet的整合External System(through Pluggable Storage)PhysicalSegmentvsegvsegvse

10、gSegment Host2Local read-only System CatalogPhysicalSegmentvsegvsegvsegSegment Host3Local read-only System CatalogPhysicalSegmentvsegvsegvsegHDFSMAGMAMagmaMagma Host1rg1 Leaderrg2 Followerrg3 FollowerTable 1 P1Table 1 P2Table 1 P3Query ExecutorMagma Host2rg2 Leaderrg1 Followerrg3 FollowerTable 1 P2Table 1 P1Table 1 P3Query ExecutorMagma Host3rg2 Followerrg3 Leaderrg1 FollowerTable 1 P2Ta