大语言模型认识与应用 课件 transformerandMOE-

Transformer前言语言模型这门课是生成式AI导……训练资料输入：人工智输出：慧输入：不要忘了今天来开输出：会输入：床前明月输出：光……Transformer找出参数论模型演进N-gramFeed-forwardNetworkRecurrentNeuralNetwork(RNN)Transformerhttps://youtu.be/dymfkWtVUdo?si=Ng29H_YxaoeiX_4yhttps://youtu.be/n9TlOhRjYoc?si=yaadpbm8w1UgbKkUhttps://youtu.be/N6aRv06iv2g?si=FuemBCZt8ChwHOvuChatGPTTransformer概述语言模型这门课是生成式AI导……论1.Tokenization2.InputLayer3.Attention4.FeedForward5.OutputLayerTransformerBlockXN文字转为Token理解Token理解上下文整合、思考…得到输出反复思考语言模型是以Token作为单位来对文字进行处理1.把文字变成TokenintroductionofgenerativeaiTokenizationintroductionofgenerativeaihttps://huggingface.co/learn/nlp-course/chapter6/5Tokenlist这页投影片没有需要训练的参数BytePairEncoding(BPE)大量文字1.把文字变成Token/tokenizer2.理解每个Token—语意introductionofgenerativeai向量(Vector)意思相近的Token会有接近的Embedding

Embeddingdogcatrabbitjumprunflowertree原本每一个Token都是独立的符号apple,cat,dot,run,jump……2.理解每个Token—语意introductionofgenerativeai

Embedding意思相近的Token会有接近的Embeddingintai198ofbank……TokenEmbedding是在训练时得到的参数查表没有考虑上下文银行岸2.理解每个Token—语意https://youtu.be/X7PH3NuYW0Q?si=XRCVhgJAfY5a8nkfWordEmbedding2.理解每个Token—语意每一个位置有一个独特的向量PositionalEmbedding……PositionalEmbedding可以在训练时得到的参数查表introductionofgenerativeai

位置1位置2位置3位置4位置5++++++3.Attention:考虑上下文Attention苹果电脑Attention来吃苹果3.Attention:考虑上下文这篇文章主要的贡献是发现不需要RecurrentNeuralNetwork(RNN)，只需要Attention就够了3.Attention:考虑上下文Attention3.Attention:考虑上下文找出相关的Token计算相关性0.1分数分数

有参数，透过训练得到计算相关性计算相关性3.Attention:考虑上下文找出相关的Token计算相关性0.1分数0.0计算相关性0.50.0计算相关性0.4AttentionWeight3.Attention:考虑上下文集合相关信息0.10.50.40.00.0(此处为简化的讲法，事实上在相加前还要过一个LinearTransformer)=0.5x0.1x0.4x++3.Attention:考虑上下文Attention3.Attention:考虑上下文计算相关性0.0计算相关性计算相关性计算相关性计算相关性0.00.40.30.3找出相关的Token3.Attention:考虑上下文0.00.00.40.30.3=0.3x0.3x0.4x++集合相关信息(此处为简化的讲法，事实上在相加前还要过一个LinearTransformer)3.Attention:考虑上下文计算所有Token两两间的相关性0.10.50.40.00.00.00.00.40.30.30.20.20.20.20.10.90.00.40.00.00.50.00.00.00.5AttentionMatrix3.Attention:考虑上下文0.30.70.0=0.7x0.3x+实作时只会考虑左边的tokenCausalAttentionMulti-headAttention关联性不只一种计算相关性0.1计算相关性计算相关性计算相关性计算相关性0.50.40.00.0计算相关性0.3计算相关性计算相关性计算相关性计算相关性0.40.00.30.0Multi-headAttention0.10.50.40.00.0=0.1x0.5x0.4x++0.30.40.00.30.0=0.3x0.4x0.3x++AttentionFeedForwardFeedForwardFeedForwardFeedForwardFeedForwardTransformerBlockTransformerBlock1TransformerBlock2Layer1Layer2…………………………TransformerBlockLastlayerOutputLayer……LinearTransformer+SoftmaxTransformer概述语言模型这门课是生成式AI导……论1.Tokenization2.InputLayer3.Attention4.FeedForward5.OutputLayerTransformerBlockXN文字转为Token理解Token理解上下文整合、思考…得到输出反复思考MixtureofExpert人类拥有复杂的认知能力2022年，YannLeCun和YoshuaBengio联合发布了NeuroAI白皮书，指出神经科学将为下一代人工智能提供指导人脑表现出稀疏的模块化结构人脑可以用更低的能耗和更灵活的机制实现复杂的认知Thesparsemodularstructureofthehumanbrain:efficiency,reusability,interpretability人脑表现出稀疏的模块化结构Sparseneuralconnectionsmake

brains

form

different

functionalmodulesLanguagecognitionrequiresthecooperation

ofdifferentmodules

in

our

brainHuman

vs.

GPT-3

Parameter:175billionNeurons:200billionActivationrate<5%Energyconsumptionperinference<15wattsEnergyconsumptionperinference:400

wattsReactionTime<100millisecondsInferenceTime:About

2000milliseconds[1]

Bartheld

et

al.,Thesearchfortruenumbersofneuronsandglialcellsinthehumanbrain:Areviewof150yearsofcellcounting.JournalofComparativeNeurology,2016.[2]

Bullmoreet

al.,

Theeconomyofbrainnetworkorganization.Naturereviewsneuroscience,2012.[3]

PeterLennie.

Thecostofcorticalcomputation.Currentbiology,2003.LargeLanguageModelsThedevelopmentofLLMs

followsahierarchicalstructureLLaMA

2LLaMA

2-chatCode

LLaMAAlignmentEnhancing

the

Coding

AbilityfLLaMA

2Enhancing

the

Ability

of

API

CallLLaMA

2-TruthXReducing

HallucinationPhind

Code

LLaMAEnhancing

the

Coding

AbilityAn

example

of

GitHubGitsupportsbranchingandmergingGeneralLLMSFTleadstoredundantmodeltraining.Incontrast,softwaredevelopmentbenefitsfromthereuseandtransfercapabilitiesMixtureofExpertsPre-trainedLanguageModelsconductsparseactivationcharacteristicsMotivation:Leveragingsparseactivationcharacteristicstoconstructpartitionsandperformbrain-likesparsecomputationWhencomputingspecificinputs,mostneuronsinFFNs

arenotactivatedMixtureofExpertsTheMixture-of-Experts(MoE)networkisarepresentativearchitectureforsparsecomputationIllustration

of

Mixture-of-Expert

(MoE)Dynamicsparseactivationrepresentasignificantdirectionforfuturemodelarchitectures.Conditionalcomputation

iskeytoscalingup.[1]

YoshuaBengio,

DeepLearningofRepresentations:LookingForward.StatisticalLanguageandSpeechProcessing,2013.[2]

Feduset

al.,

SwitchTransformers:ScalingtoTrillionParameterModelswithSimpleandEfficientSparsity.JMLR,2022.MixtureofExpertsEachFFNmoduleisregardedasan“Expert”EachexpertisdesignedtohandlespecifictypesoftasksordatasubsetsSparseMoElayers

are

used

to

replace

the

Feed-Forward

Network

(FFN)

Layer[1]

Feduset

al.,

SwitchTransformers:ScalingtoTrillionParameterModelswithSimpleandEfficientSparsity.JMLR,2022.MixtureofExpertsTherouterdetermineswhichtokensaresenttowhichexpert[1]

Feduset

al.,

SwitchT