ArchitectureforBigDataOrca一个处理大数据的模块化的

Orca:AModularQueryOptimizerArchitectureforBigData

Orca:一个处理大数据的模块化的查询优化体系Sigmod2014报告人万丽蓉研究背景InthispaperwepresentthearchitectureofOrca,thenewqueryoptimizerforallPivotal（Pivotal公司）datamanagementproducts,includingPivotalGreenplumDatabase(GPDB)andPivotalHAWQ.GPDBarchitectureGPDBarchitecture工作原理储存和处理如此大量的数据是通过将负载分布到几个独立的服务器或主机，创建一系列个体数据库，一起协作，提供一个单一的数据库映像。主管服务器（mastersevers）是GPDB的入口，也就是用户连接数据库提交SQL语句的地方。主管服务器负责协调与其他数据库实例共同工作，处理和存储数据。每一个数据库实例成为一个segment。当一个查询被提交给主管服务器，查询将被优化并分解为更加细化的任务，分布到每一个segment进行处理，共同提交最终结果。网络层负责segments之间的进程进行通讯。

HAWQHAWQ,amassivelyparallelSQL-compliantengineontopofHDFS（Hadoop分布式系统）.HAWQ在内核中使用Orca体系设计高效的查询方案，降低Hadoop集群的数据访问成本。HAWQ把最新的基于成本的优化和Hadoop的可扩展性容错性结合了起来，在PB级规模上实现了数据的交互处理。InteractionofOrcawithdatabasesystemOrcaisthenewqueryoptimizerforPivotaldatamanagementproducts,includingGPDBandHAWQ.DXL：查询优化器的解耦需要建立一个沟通机制与数据库系统通信来处理查询。Orca体系包括一个框架，用于数据库系统和优化器之间的数据交换，这个框架叫做DataeXchangeLanguage（DXL）。InteractionofOrcawithdatabasesystemtranslatorInteractionofOrcawithdatabasesystemFigure2showstheinteractionbetweenOrcaandanexternaldatabasesystem.TheinputtoOrcaisaDXLquery.TheoutputofOrcaisaDXLplan.Duringoptimization,thedatabasesystemcanbequeriedformetadata(e.g.,tabledefinitions).Orcaabstractsmetadataaccessdetailsbyallowingdatabasesystemtoregisterametadataprovider(MDProvider)thatisresponsibleforserializingmetadataintoDXLbeforebeingsenttoOrca.Thedatabasesystemneedstoincludetranslatorsthatconsume/emitdatainDXLformat.Query2DXLtranslatorconvertsaqueryparsetreeintoaDXLquery,whileDXL2PlantranslatorconvertsaDXLplanintoanexecutableplan.TheimplementationofsuchtranslatorsisdonecompletelyoutsideOrca,whichallowsmultipletouseOrcabyprovidingtheappropriatetranslators.OrcaarchitectureMemo.Thespaceofplanalternativesgeneratedbytheoptimizerisencodedinacompactin-memorydatastructurecalledtheMemo.TheMemostructureconsistsofasetofcontainerscalledgroups,whereeachgroupcontainslogicallyequivalentexpressions.Memo是一个由group的集合。每一个group包含一类逻辑等价式。

Memogroupscapturethedifferentsub-goalsofaquery(e.g.,afilteronatable,orajoinoftwotables).将查询拆解成子目标，由group获得。Groupmembers,calledgroupexpressions,achievethegroupgoalindifferentlogicalways(e.g.,differentjoinorders).Eachgroupexpressionisanoperatorthathasothergroupsasitschildren.SearchandJobSchedulerOrcausesasearchmechanismtonavigatethroughthespaceofpossibleplanalternativesandidentifytheplanwiththeleastestimatedcost.Thesearchmechanism由专门的JobScheduler来创建独立或者并行的workunits来完成，大概分为三个步骤：

explorationwhereequivalentlogicalexpressionsaregenerated,implementationwherephysicalplansaregenerated,andoptimization,whererequiredphysicalproperties(e.g.,sortorder)areenforcedandplanalternativesarecosted.TransformationsPlanalternativesaregeneratedbyapplyingtransformationrulesthatcanproduceeitherequivalentlogicalexpressions(e.g.,InnerJoin(A,B)→InnerJoin(B,A)),orphysicalimplementationsofexistingexpressions(e.g.,Join(A,B)→HashJoin(A,B)).Theresultsofapplyingtransformationrulesarecopied-intotheMemo,whichmayresultincreatingnewgroupsand/oraddingnewgroupexpressionstoexistinggroups.Eachtransformationruleisaself-containedcomponentthatcanbeexplicitlyactivated/deactivatedinOrcaconfigurations.PropertyEnforcementOrcaincludesanextensibleframeworkfordescribingqueryrequirementsandplancharacteristicsbasedonformalpropertyspecifications.Propertieshavedifferenttypesincludinglogicalproperties(e.g.,outputcolumns),physicalproperties(e.g.,sortorderanddatadistribution),andscalarproperties(e.g.,columnsusedinjoinconditions).Duringqueryoptimization,eachoperatormayrequestspecificpropertiesfromitschildren.Anoptimizedchildplanmayeithersatisfytherequiredpropertiesonitsown(e.g.,anIndexScanplandeliverssorteddata),oranenforcer(e.g.,aSortoperator)needstobepluggedintheplantodelivertherequiredproperty.Theframeworkallowseachoperatortocontrolenforcersplacementbasedonchildplans'propertiesandoperator'slocalbehavior.MetadataCacheSincemetadata(e.g.,tabledefinitions)changesinfrequently,shippingitwitheveryqueryincursanoverhead.Orcacachesmetadataontheoptimizersideandonlyretrievespiecesofitfromthecatalogifsomethingisunavailableinthecache,orhaschangedsincethelasttimeitwasloadedinthecache.Metadatacachealsoabstractsthedatabasesystemdetailsfromtheoptimizer,whichisparticularlyusefulduringtestinganddebu