面向NetGPT的语义通信开放性问题

第0第0页面向NetGPT的语义通信开放性问题张宏纲李荣鹏,陈先福,赵志峰(1)之江实验室,(2)浙江大学,(3)VTT2023年5月21日-鹏城实验室Transformer-based?!Transformer-based?!Transformer-based!!ZhilinTransformer-based?!Transformer-based?!Transformer-based!!ZhilinLu,RongpengLi,K.Lu,XianfuChen,EkramHossain,ZhifengZhao,andHonggangZhang,tionATutorialcumSurveyapsarxivorgabsDec2022.1nticG基于注意力机制的Transformer结构与算法NShazeeretalAttentionIsAllYouNeedstConferenceonNeuralInformationProcessingSystems(NIPS2017),CA,USA,2017.2注：图片来自网上下载23注：图片来自网上下载34注：图片来自网上下载4GNetGPTGPTNet&Net4GPT)Crowcroft,andHonggangDesign,Architectures,andJianjunWu,RongpengLi,XueliAn,Crowcroft,andHonggangDesign,Architectures,and5Zhang,“TowardNativeArtificialIntelligencein6GNetworks:System5ParadigmsarXiv2823,March2021.GNetGPT(GPT4Net&Net4GPT)JianjunWu,RongpengLi,XueliAn,ChenghuiPeng,ZheLiu,JonCrowcroft,andHonggang6Zhang,“TowardNativeArtificialIntelligencein6GNetworks:SystemDesign,Architectures,andParadigmsarXiv2823,March2021.6开放性问题:如何基于通信网络(云网边端一体化)构建分布式Transformer的自适应预训练大模型架构(分级分层)？7注：图片来自网上下载7开放性问题:如何基于通信网络(云网边端一体化)构建分布式Transformer的自适应预训练大模型架构(分级分层)？8：图片8来自网上参考案例:NVIDIAMegatron-超大Transformer言模型的分布练框架 (模型切分)①大规模分布式训练是训练大模型的必须②NVIDIAMegatron是开源的、软硬协同设计的训练框架，专为Transformer-based的超大语言模型设计③NVIDIADGXSuperPOD是开放的集群参考设计架构，专为大规模分布式训练而准备④Megatron优化的Tensor模型并行：用于Intra-Transformer层，可以高效执行在HGXbased的系统上⑤Megatron优化的Pipeline模型并行：用于Inter-Transformer层，可以有效利用集群中多网卡的设计。⑥数据并行的加持，可以扩展到更大规模、训练更快⑦GPT-3175B的大模型，在1024张A100上可达到44%左右的计算效率9注：图片来自网上下载9参考案例:NVIDIAMegatron-超大Transformer语言模型的分布式训练框架(并行、分布式/分层/分级组网)10注：图片来自网上下载10参考案例:百度飞桨分布式自适应预训练大模型架构注：图片来自网上下载11Envisionedend-to-endsemanticcommunicationandnetworkingarchitectureEnvisionedend-to-endsemanticcommunicationandnetworkingarchitectureAddressing,Routing&Switching(网络语义使能)RongpengLi,ZhifengZhao,XingXu,FeiNi,andHonggangZhang,“TheCollectiveAdvantageforAdvancingCommunicationsandIntelligence,”IEEEWirelessCommunications,June2020.ElifUysal,OnurKaya,etal.,“SemanticCommunicationsinNetworkedSystems:ADataSignificancePerspective,”IEEENetwork,July/August2022.12注：图片来自网上下载12Multi-HeadAttentionModuleraphNeuralNetworkMulti-HeadAttentionModulePopularityPredictionandContentCaching(分布式预训练)TemporalGAT:TemporalGraphAttentionNetworkAoI-basedtemporalattentiongraphneuralnetworkandthemulti-headattentionmechanismJianhangZhu,RongpengLi,GDing,ChanWang,JianjunWu,ZhifengZhao,andHonggangZhang,“AoI-basedTemporalAttentionGraphNeuralNetworkforPopularityPredictionandContentCachingIEEETransCognitiveCommunicationsandNetworkingTCCN,December2022.13In\?IIi6?n\Domdu\in6,eSDi?nD?deJ\n?JrouJne