大规模图神经网络系统架构剖析

1、 扬帆起航AliGraph: 大规模图神经网络系统架构剖析Analysis of Large-scale GNN System Architecture Based on AliGraphContents 目录01 AliGraph平台简介02 系统架构与核心技术AliGraph平台简介AliGraph Overview01PAI：阿里巴巴机器学习平台Platform of Artificial Intelligence in AlibabaPAI机器学习平台特性Features of PAI/product/learn一站式机器学习平台数据处理，特征加工 模型训练，在线预测SOTA机器学习算

2、法经典机器学习 阿里自研算法 兼顾离线&实时精心优化的深度学习引擎阿里内部大规模任务打磨PAI-Tensorflow引擎弹性模型预测服务简化模型部署负担 支持扩缩容，降低成本什么是图神经网络Introduction of GNNGNN低维向量空间G = (V, E V=VertexE=EdgeNeura, Network实践中的挑战Challenges in Practice数据不规则稀疏性，非结构化，属性 同构，异构，有向，无向Data Cleaner规模巨大千亿边、百亿点 样本指数级膨胀百TB级万CPU时计算难扩展图算子种类变化多 无法结合深度学习用户期望：一体化解决方案Users Exp

3、ect Solutions from Data to Model for GNNAPPGNN AlgorithmsGraph & TensorEngineGraph DataLarge-scale ,Homogeneous ,Heterogeneous,AttributedAliGraph：云上的“图+深度学+”平台A GNN Platform on Alibaba Cloud支持PB级 源于实际 生产的复杂图数据企业级Industry Level简洁的IDEI便于从原始 数据构建业务模型端到端End to End满足决速发展的研究与 实际生产应用的结合产与研For Research and

4、Production系统架构与核心技术System Architecture and Key Technologies02GNN编程范式GNN Programming Paradigmhop=1hop=21. Sampling2. Vectorization3. Aggregate4. CombineVertex IdVertex AttributesEdge AttributesEdge IdAggregate outputVertex outputV1V2E1E2V0一体化系统实现An Integrated System Implementation for GNNServer 0Serv

5、er Server 2Server nTensor EngineGraph EngineData BridgePythonGraph QuerySamplerTensorFlow & PyTorchAggregatorCombinerLarge-scaleHeterogeneousAttributedGNNRPCFriendlyFlexibleRobustEffective简洁的用户接口Laconic User Interface点击相似高效的图构建过程Effective Graph BuildingSERVER iData QueueData QueueData QueueLoca lPRO

6、CESSING THREADPROCESSING THREADPROCESSING THREADData QueueData QueueData QueuePROCESSING THREADPROCESSING THREADPROCESSING THREADLOADING THREADRPCRPCLoca lLOADING THREADLOADING THREADRPCRPCSERVER jLOADING THREADBatch ReadingBatch ReadingPartitioningPartitioningPartitioningPartitioningBatch ReadingBa

7、tch ReadingGraph ParsingGraph ParsingGraph ParsingGraph ParsingGraph ParsingGraph ParsingUpdate StorageUpdate StorageUpdate StorageUpdate StorageUpdate StorageUpdate Storage多跳采样加速机制Cache to Empower Multi-Hop Graph Sampling可扩展的算子类型Flexible and Pluggable OperatorsUser nterfaceDistribute RuntimeDistribute StorageProgramming nterfaceProgramming nterfaceOperatorOperatorOperatorOperator系统数据System Perfo