大模型核心技术与应用（微课视频版）课件 Transformer

Transformer0.翻译问题如何用神经网络解决中文到英文的翻译问题？1.如何用数字表示中文和英文。2.如何让神经网络理解语义。3.如何让神经网络生成英文。1.Tokenize我爱吃苹果

我|爱|吃|苹果我喜欢苹果手机

我|喜欢|苹果|手机1.Embedding我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68可学习参数1.Embedding航母飞机飞艇大小重量坦克汽车1.Embedding航母飞机苹果坦克橘子香蕉筷子盘子杯子排球篮球足球1.Embedding皇后男人国王女人

2.SelfAttention我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68我喜欢手机苹果1.12-0.42...-0.421.32-0.620.13-1.22...-0.120.42-0.420.121.15...-0.820.791.250.89-1.12...-0.780.24-0.682.SelfAttention我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68我喜欢手机苹果1.12-0.42...-0.421.32-0.620.13-1.22...-0.120.42-0.420.121.15...-0.820.791.250.89-1.12...-0.780.24-0.682.QKV我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.681.120.75...-0.321.220.82-0.75...1.220.250.121.15...1.320.72-1.620.95...-0.322.011.821.25...1.721.251.521.15...1.521721.120.55...-0.320.621.82-0.75...1.221.250.721.85...1.520.520.120.75...1.32-0.220.82-0.85...1.225.250.525.15...2.320.72

2.SelfAttention我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.681.120.75...-0.321.220.82-0.75...1.220.250.121.15...1.320.72-1.620.95...-0.322.011.821.25...1.721.251.521.15...1.521721.120.55...-0.320.621.82-0.75...1.221.250.721.85...1.520.520.120.75...1.32-0.220.82-0.85...1.225.250.525.15...2.320.72

2.MultiHeadAttention苹果0.89-1.12...-0.780.24-0.681.120.75...-0.321.220.82-0.75...1.220.250.121.15...1.320.72

1.120.75...-0.321.220.82-0.75...1.220.250.121.15...1.320.72

...类似于卷积神经网络里的多个卷积核2.MultiHeadSelfAttention我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.681.120.75...-0.321.220.82-0.75...1.220.250.121.15...1.320.72-1.620.95...-0.322.011.821.25...1.721.251.521.15...1.521721.120.55...-0.320.621.82-0.75...1.221.250.721.85...1.520.520.120.75...1.32-0.220.82-0.85...1.225.250.525.15...2.320.72

Q0.120.75...1.32-0.221.120.55...-0.320.62-1.620.95...-0.322.011.120.75...-0.321.22K0.82-0.85...1.225.251.82-0.75...1.221.251.821.25...1.721.250.82-0.75...1.220.25V0.525.15...2.320.720.721.85...1.520.521.521.15...1.521720.121.15...1.320.72

2.MultiHead实现

Q0.120.751.32-0.221.120.55-0.320.62-1.620.95-0.322.011.120.75-0.321.22K0.82-0.851.225.251.82-0.751.221.251.821.251.721.250.82-0.751.220.25V0.525.152.320.720.721.851.520.521.521.151.521720.121.151.320.725.152.321.851.521.151.521.151.320.82-0.851.82-0.751.821.250.82-0.750.720.521720.720.821.821.820.82Head1Head2Head3

2.代码实现3.LayerNormalization我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68对每一个Token都会独立计算均值和方差，进行标准化。3.LayerNormalization我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68对每一个Token都会独立计算均值和方差，进行标准化。

3.代码实现3.残差连接sublayerx

4.FeedForwardTransformer整体架构PositionEncoding我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.681.120.75...-0.321.220.82-0.75...1.220.250.121.15...1.320.72-1.620.95...-0.322.011.821.25...1.721.251.521.15...1.521721.120.55...-0.320.621.82-0.75...1.221.250.721.85...1.520.520.120.75...1.32-0.220.82-0.85...1.225.250.525.15...2.320.72

Attention机制没有考虑位置信息PositionEncoding我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.6800...00100...01000...01100...100PositionEncoding我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.6800...00100...01000...01100...100PositionEncoding

[0.95,0.87,0.42][-0.78,0.91,0.78]Token1Token2PositionEncoding我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68

...

...

...

...

PositionEncoding我爱吃苹果1.12-0.42...-0.421.32-0.620.12-1.42...-0.220.32-0.520.521.45...-0.421.32-0.620.89-1.12...-0.780.24-0.68

...

...

.