2024年AI+研发数字峰会(AiDD峰会)：SUBLLM新架构：文本下采样机制革新大语言模型效率

A:nnAl+研发数字峰会2024Al+研发数字峰会Al+DevelopmentDigitalsummitAI驱动研发变革促进企业降本增效北京站08/16-17王全东小米大模型团队Al+研发数字峰会Al+DevelopmentDigitalsummitAl+Al+DevelopmentDigitalsummitAl+研发数字峰会Al+DevelopmentDigiAl+研发数字峰会Al+DevelopmentDigitalsummit长文本模型的技术挑战行业报告新闻摘要长文写作行业报告新闻摘要长文本模型的技术挑战长文本模型结构：Al+研发数字峰会Self-Attention(Groupe长文本模型的技术挑战5~10BtokensAl+研发数字峰会Al+DevelopmentDigitalsummitlILⅢl1k-2k凶16k-32k长文本模型的技术挑战Al+研发数字峰会Al+DevelopmentDigitalsummit长文本模型的技术挑战Al+研发数字峰会Al+DevelopmentDigitalsummitPoSE:2048tokens0……0……45752Full-PoSE:2048tokens0……0……4575201…81918k窗长2~15Ttokens5~10Btokens5~10Btokens0长文本模型的技术挑战DecoderonlyTransformer长文本模型训练infra开发优势：对Attention的实现不敏感，适合各种attention方法劣势：序列并行度不能超过头数Al+研发数字峰会Al+DevelopmentDigital[N,d](Local:[N/P,dj)alltoallcommX[N,d/P]1[N,d]([N,d/P]1[N,d](Loca:[N/P,dj)matmulQ[N,d](Q[N,d](Local:[N/P,dl)[d,N](Local[d,N](Local:[d,N/P)[d/P,N]K[d/P,N]K33KTK[N,d](Local:[N/P,d])K[N,d](Local:[N/P,d])matmulmatmulVValltoallcomm1QnK1QnKsoftmax[N,d/P]/[N,d/P]/PP[N,d/P]N:sequencelengthd:hiddensizehc:headcountP:totalprocessor(GPU)countAssumesP=hc=4matmulW₀[N,d](Local:[N/Pdj)Al+DevelopmentDigitalsummit长文本模型的技术挑战Al+DevelopmentDigitalsummit优势：并行度的扩展性较好劣势：对Attention变种不友好，eg.SparseAttentionBlockwiseBlockwiseK:dxN[FedForwardFeedForwardCopyBlocktoSRAMKeyValueBlockBlockwiseKeyValueBlockBlockwiseKeyValueBlockQ:NxdOuterLoopV:NXdAtentionAtentionqueryblockqueryblockDevice1Device2(b)Blockwise_BlockwiseBlkiCopyonSRAMQuery1BlockwiseBlockwiseBlockwiseBlockwiseAttentionAttentionAttentionAttentionQuery2Key1Key2Key3Key4Query3Value1Value2Value3Value4sm(QKTV:N×d[Query4compute,sendtoreceivefromnextdevicepreviousdeviceInnerLoopKeyandValuelnnerLoop长文本模型的技术挑战推理成本高：attention的平方复杂度推理速度角度，200k比20k贵28倍，比2k贵112倍Al+研发数字峰会Al+DevelopmentDigitalsummit长文本模型的技术挑战其他长文本模型结构：Al+研发数字峰会Al+DevelopmentDigitalsummitLinearprojectionConcatConcatRetrieveattention&PEUpdate{KV}s-1{KV}s其他长文本模型结构：NAl+研发数字峰会Al+研发数字峰会2.MEGALODON:继承MEGA(带有门控注意力的指数移动平均)并改进由H古OQ&KVCEMAoutputx′LayerinputXAttention×AttentionTimestepNorm×(a)SketchofMegalodonlayer.(b)Pre-Norm(c)Two-hopResidualMEGALODON模型结构byMeta,2024推理加速based大模型兼容主流attention预训练大模型：eg.Llama,可持续训练投机解码√×√√×√√×√√×√动态稀疏attention:eg.√×√√×√√×√√√√√√√×××√√×××√√√××√√√√√×√√√√√×√√√√√√√×√√√√√×√√√√√√Al+Al+DevelopmentDigitalsummit1.开发一种优化资源使用的架构，同时保持模型能力不变。2.区分重要token和不重要token,重要token占主要算力3.兼容现有attentionbased模型生态，模型广泛应用的关键Al+研发数字峰会Al+DevelopmentDigitalsummit受语音领域启发：语音信号下采样减少冗余保留必要信息语音识别中语音分帧后降采样4倍LayernormConformerBlocksxN|DropoutLinearFeedForwardModule+ConvolutionModuleConvolution+Subsampling10msrateMulti-HeadSelfAttentionModuleSpecAug10msrate+FeedForwardModule1111iAl+研发数字峰会Al+DevelopmentDigitalsummit5050Hz50Hz25Hz→*25Hz12.5Hz50Hz50Hz50Hz50Hz50Hz;100Hz25Hz50Hz2×2×卡十十→→→→Zipformer模型结构byDanielPovey,2024受语音领域启发：语音信号下采样减少冗余保留必要信息文本序列是否存在冗余?Yes!Up-samplePoolPoolAl+DevelopmentDigitalsummitAl+DevelopmentDigitalsummitBlock1Block2Block3EncoderDecoder(受语音领域启发：语音信号下采样减少冗余保留必要信息文本序列是否存在冗余?Yes!Al+研发数字峰会Al+研发数字峰会nnToclassificationhead—ABlock3SoftmaxClassProbabilitiesForEncoder-OnlySettingTruncateForEncoder-DecoderSetting不SpectralFilterUpsample+Upsample÷+Decoder十Residual—受语音领域启发：语音信号下采样减少冗余保留必要信息文本序列中的token是否同等重要?No!LightMLPHeavyMLPmRoutingAl+研发数字峰会Al+DevelopmentDigitalsummitLightAttentionHeavyAttentionRoutingq,vHeavybranch受语音领域启发：语音信号下采样减少冗余保留必要信息SUBLLM:采样(Sub-sampling)-上采样(Up-sampling)-旁路(Bypass)LLMBypassConnection_1BypassConnection_2LUpsampling_2●●BBssapyBBsBssapyBUpsampling_1/pdf/2406.06571,acceptedbyECAI2024受语音领域启发：语音信号下采样减少冗余保留必要信息BlocksModelrepresentation122Al+Al+研发数字峰会Al+DevelopmentDigitalsummit1.通过去除不太重要的tokens来缩短序列长度2.使用Score层来衡量token的重要性x=INDEXSELECT(x,I)x=INDEXSELECT(x,I)Score层给每个tokenScore层给每个token打分再得到weight=S(x1),...,S(xn)W=W1,...,WNWn=CLAMP(BALANCER(sn,[0,1]))I={i|wiI={i|wi∉TOPK(w,N')}LearnableSubsamplingModule妙处1:解决训练和推理的不一致性位置编码下采样RelPos(x{,xj)=RoPE(x;,xj,(Ii-I;))Al+研发数字峰会Al+DevelopmentDigitalsummitInferenceMode:推理时看不到未来token,设定阈值为0,正数分值的token保留Iinfer={i|wi≥v}Balancer:通过对梯度进行惩罚来限制score的正数比例=预设下采样保留比例Al+研发数字峰会Al+研发数字峰会Al+DevelopmentDigitalsummitWkept=INDEXSELECT(w,T)Wdiscarded=INDEXSELECT(w,)Wsample;～UNIFORM(E),fori=1,2,…,N′Wscaling=Wkept—WsampleWsample实际基本为0,不影响weight大小，但是有梯度otherwiseAl+DevelopmentDigitalsummit1.下采样前和上采样后的序列加权求和，perchannel2.增强训练的收敛性和稳定性。Al+研发数字峰会Al+DevelopmentDigitalsummitSUBLLM-BypassModule+ResidualConnection-BypassModuleLLaMA1.给重要token分配更多算力2.MoD每间隔一个block做一次topk的token筛选3.加速50%Mixture-of-DepthsRoutingDecisionsSequenceEarly-ExitVanillaTransformerMixture-of-DepthsRoutingDecisionsSequenceEarly-ExitVanillaTransformerSequenceSequencelayersNormalizeSelf-attentionNormalizeTw=0.65□hRouteRouteSelf-attentionNormalize个Normalizew=0.41Al+研发数字峰会Al+DevelopmentDigitaAl+Al+DevelopmentDigitalsummit主要实验结果Al+研发数字峰会Al+DevelopmentDigitalAl+研发数字峰会Al+DevelopmentDigitalsummitBF16,FlashAttention2主要实验结果Al+研发数字峰会Al+DevelopmentDigitalsummitEfficiencySpeed-UpMaxSpeed-UpInferenceTGS个Ratio个Mem(GB)↓TGS个Ratio个Speed个Ratio个Mem(GB)↓65.99SST2个Amazon个DBpediaYelp个Hate↑Avg.个81.0186.5442.9764.7787.5968