ClickHouse和他的朋友们（2）MySQL Protocol和Read调用栈VIP

1、ClickHouse和他的朋友们（2）MySQL Protocol和Read调用栈作汨个OLAP的DBMS来说有2个端非常重要：用尸钾1何方便的钓殡来，这是入口端-ClickHouse除了自己的client外，还提供了 MySQL/PG/GRPC/ 等接入方式数据如何方便的挂上去，这是数据源端 ClickHouse除了自己的引擎外，还可以挂载MySQL/Kafka等外部数据源这不油外互通，多条朋友多条路，以以改据”级的编排能力今天送的是入口端的MySQL十舜N也是本系列ClickHouse的第一个好朋友，用尸可通过 MySQL客万中或相关Driver直接钓接至U ClickHouse,进行数据

2、谆等操作.本文通过MySQL的QueryiW,借用渔杉g了解下ClickHouse的数据全过程如何实班L入口文件在：MySQLHandler.cpp握手协议MySQLCIientGreeting 数据中良到 MySQLHandlerMySQLHandler 回复一个 Greeting-Response 丰度MySQLCIient 合MySQLHandler又蒙疝琅进行茎霞，并返回皿结果MySQL Protocol Core/MySQLProtocol.hQuery iW当为西过后，就可以进行正常的数据交互了当 MySQLCIientmysql SELECT * FROM system.numb

3、ers LIMIT 5;MySQLHandler 的调木其-MySQLHAndler:comQuery - executeQuery - pipeline-execute - MySQLOutputFormAt:coMySQLCIient接收至I结果在步马executeQuery(executeQuery.cpp)非常重要它是所有前端Server和 ClickHouse内核的接入口，第一个参数是SQL文本(selectl),第二个参数是结果集要芨 到哪里去(socket net)调用栈分析SELECT * FROM system.numbers LIMIT 5I.获取数据源StorageSys

4、temNumbers 数据源：DB:StorAgeSystemNumbers:reAd(std:_1:vectorstd:_1:bAsic_stringchArz std:_1:DB:ReAdFromStorAgeStep:ReAdFromStorAgeStep(std:1:shAred_ptrDB:RWLocklmpl:Lock DB:InterpreterSelectQuery:executeFetchColumns(DB:QueryProcessingStAge:Enum, DB:Qu DB:InterpreterSelectQuery:executeFetchColumns(DB:Qu

5、eryProcessingStAge:Enumz DB : : Qu DB:InterpreterSelectQuery:executelmpl(DB:QueryPlAn&, std:_1:shAred_ptrreAd(required_columns, metAdAtA_snApshot, query_info, *context,Pipeline构造unsigned long, unsigned long, unsDB:LimitTrAnsform:LimitTrAnsform(DB:Block const&zDB:Limitstep:trAnsformPipeline(DB:QueryP

6、ipelines) memory:2214DB:Limitstep:trAnsformPipeline(DB:QueryPipelines) memory:2299DB:Limitstep:trAnsformPipeline(DB:QueryPipeline&) memory:3570DB:LimitStep:trAnsformPipeline(DB:QueryPipelines) memory:4400Limitstep.cpp:33DB:Limitstep:trAnsformPipeline(DB:QueryPipe1ine&)DB:ITrAnsformingStep:updatePipe

7、line(std:1:vectorstd:1:unique_ptrDB:QueryPDB:QueryPlAn:buildQueryPipeline() QueryPlAn.cpp:154DB:InterpreterSelectWithUnionQuery:execute() InterpreterSelectWithUnionQuery.cpp:200 DB:executeQueryImpl(const chAr *, const chAr *, DB:Context &, bool, DB:QueryProcessDB:executeQuery(DB:ReAdBuffer&, DB:Writ

8、eBuffer&z bool, DB:Context&, std:1:funcDB:MySQLHAndler:comQuery(DB:ReAdBuffer&) MySQLHAndler.cpp:307DB:MySQLHAndler:run() MySQLHAndler.cpp:141Pipeline主施DB:LimitTrAnsform:prepAre(std:1:vectorunsigned long, std:1:AllocAtorunsignDB:PipelineExecutor:prepAreProcessor(unsigned long, unsigned long, std:1:q

9、ueueD DB:PipelineExecutor:tryAddProcessorToStAcklfUpdAted(DB:PipelineExecutor:Edges, std DB:PipelineExecutor:prepAreProcessor(unsigned long, unsigned long, std:1:queueD DB:PipelineExecutor:tryAddProcessorToStAckIfUpdated(DB:PipelineExecutor:Edges, std DB:PipelineExecutor:prepAreProcessor(unsigned lo

10、ng, unsigned long, std:1:queueDDB:PipelineExecutor:tryAddProcessorToStAckIfUpdAted(DB:PipelineExecutor:Edges, std DB:PipelineExecutor:prepAreProcessor(unsigned long, unsigned long, std:1:queueD DB:PipelineExecutor:tryAddProcessorToStAckIfUpdAted(DB:PipelineExecutor:Edges, std DB:PipelineExecutor:pre

11、pAreProcessor(unsigned long, unsigned long, std:1:queueDDB:PipelineExecutor:tryAddProcessorToStAckIfUpdAted(DB:PipelineExecutor:Edge&, std DB:PipelineExecutor:prepAreProcessor(unsigned long, unsigned long, std:1:queueDDB:PipelineExecutor:initiAlizeExecution(unsigned long) PipelineExecutor.cpp:747 DB

12、:PipelineExecutor:executelmpl(unsigned long) PipelineExecutor.cpp:764DB:PipelineExecutor:execute(unsigned long) PipelineExecutor.cpp:479DB:executeQuery(DB:ReAdBuffer&, DB:WriteBuffer&z bool, DB:Contexts, std:1:funcDB:MySQLHAndler:comQuery(DB:ReAdBuf fer&) MySQLHAndler.cpp:3 07DB:MySQLHAndler:run() M

13、ySQLHAndler.cpp:141Output打比发送DB : :MySQLOutputForinAt: : consume (DB : :Chunk) MySQLOutputFormAt. cpp : 53DB:lOutputFormAt:work() lOutputFormAt.cpp:62DB:executeJob(DB:IProcessor *) PipelineExecutor.cpp:155operAtor() PipelineExecutor.cpp:172DB:PipelineExecutor:executeStepImpl(unsigned long, unsigned

14、long, std:1:AtomicbDB:PipelineExecutor:executeSingleThreAd(unsigned long, unsigned long) PipelineExecutDB:PipelineExecutor:executelmpl(unsigned long) PipelineExecutor.cpp:812DB:PipelineExecutor: :execute(unsigned long) PipelineExecutor.cpp:4 7 9DB:executeQuery(DB:ReAdBuffer&, DB:WriteBuffer&z bool,

15、DB:Context&, std:1:funcDB:MySQLHAndler:comQuery(DB:ReAdBuf fer&) MySQLHAndler.cpp:311DB: :MySQLHAndler: :run () MySQLHAndler.cpp:141总结ClickHouse的模块化比拟清晰，像弟积木一木革以组合拼装，当我。拌t：SELECT * FROM system.numbers LIMIT 5首先内核解析SQL用生成AST,然后根据AST获取数据源Source, pipeline.Add(Source)其次根据AST信息生成QueryPlan,根据QueryPlan再生成相应 的Transform, pipeline.Add(LimitTransform)然后添加 Ou卬ut Sink 作7数据名又续, pi pel ine.Add (OutputSink) pipeline,各个 Transformer 开女台工作ClickHouse的Transformer y用系统叫做Processor,也是决定性能的重要模块，诅# 口 Pipeline处