2024微软人工智能系统深度学习中的分布式训练算法

单一设备Top-5accuracyvs.computationalcomplexity(singleforwardpass)BenchmarkAnalysisofRepresentativeDeepNeuralNetworkArchitectures/pdf/1810.00736.pdf模型相关，相对固定 可变因素可变因素工作重点Moore定律，可变因素工作重点相对有限本章内容本章内容下章内容 TensorFlow

Data-FlowGraph(DFG)x=tf.placeholder(tf.float32)W=x=tf.placeholder(tf.float32)W=b=tf.Variable(tf.float32)m=W*xs=m+by=tf.reduce_sum(s)grad_W,grad_b=tf.gradients(y,[W,b])update=optimizer.apply_gradients({[W,grad_W],[b,grad_b]})xWb𝛻W𝛻b*m*𝐠++𝐠𝛻ms𝛻sΣΣ𝐠y x=tf.placeholder(tf.float32)W=b=tf.Variable(tf.float32)

多个样本

Data-FlowGraph(DFG)m=W*xs=m+by=tf.reduce_sum(s)grad_W,grad_b=tf.gradients(y,[W,b])update=optimizer.apply_gradients({[W,grad_W],[b,grad_b]})

x W *m

𝛻W 𝛻b*𝐠𝛻m+ +𝐠s

多个操作Σ y

张量计算算子间并行数据并行：多个样本并行执行模型并行：多个算子并行执行组合并行：多种并行方案组合叠加https://blog.skymind.ai/distributed-deep-learning-part-1-an-introduction-to-distributed-training-of-neural-networks/ im2colconvolution im2colconvolution JosephE.GonzalezAI-SystemsDistributedTraining算子内并行算子并行：并行单个张量计算子内的计算（GPU多处理单元并行）https://blog.skymind.ai/distributed-deep-learning-part-1-an-introduction-to-distributed-training-of-neural-networks/AllReducex W b *

AllReduce𝛻W 𝛻b*𝐠

AllReducex W b*

AllReduce𝛻W 𝛻b*𝐠m +

𝛻m

m+

𝛻ms 𝛻s

s 𝛻sΣ y

GPU0

Σ y

GPU1xWb𝛻W𝛻b*m*𝐠+xWb𝛻W𝛻b*m*𝐠++𝐠𝛻mGPU0s𝛻sΣΣ𝐠yGPU1非并行数据并行模型并行样本数据量11/N1传输数据量0模型大小激活大小存储占用1N1负载平衡度-强弱并行限制-单步样本量算子数量具有同步障不含同步障具有限定的宽松同步障ComputeCommunicateComputeIterationComputeCommunicateComputeIterationw1wIterationw2wIterationw3BarrierwBarrierWasteIterationWasteIterationWasteIterationwwMachine2wwMachine3wwJosephE.GonzalezAI-SystemsDistributedTrainingComputeCommunicateComputeIterationComputeCommunicateComputeIterationIterationIterationWasteIterationWasteIterationWasteIterationMachine2Machine3Barrier BarrierJosephE.GonzalezAI-SystemsDistributedTrainingMachine1Machine2

Compute Communicate

ComputeIterationIterationIterationIterationIterationIterationIterationIterationJosephE.GonzalezAI-SystemsDistributedTraining具有同步障AAA不含同步障