大模型核心技术与应用（微课视频版）课件 大模型强化学习

大模型强化学习SFTRewardPPORewardModelRewardModel数据用户偏好数据不怕不识货，就怕货比货questionchosen数据库是一个有组织的数据集合，允许高效的数据存储、检索和管理。rejected数据库用于存储数据。什么是数据库？RewardModel模型LLM什么是数据库？数据库用于存储数据。LMHead字典维度

LLM什么是数据库？数据库用于存储数据。ValueHead得分

RewardModelLoss

RewardModelTrainerPPOPPO数据PPO模型Reference

LLMActiveLLMRewardLLMState-ValueModel基准模型训练模型奖励模型状态价值模型LMHead(HS*VS)HS:hidden_sizeVS:vocab_sizeLMHead(HS*VS)ScoreHead(HS*1)ValueHead(HS*1)LMHead(HS*VS)Reference

LLM基准模型训练模型+状态价值模型奖励模型LMHead(HS*VS)HS:hidden_sizeVS:vocab_sizeScoreHead(HS*1)ValueHead(HS*1)为什么Score

天空

蓝色

是

？

光

因为

散射

的

。0000000003.8Prompt+Response基础模型[0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...]训练模型[0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...][0.11,0.02,0.02,0.32,0.04...]RewardKL0.51.23.62.34.84.51.55.31.02.1Reward-0.1-0.24-0.72-0.46-0.96-0.9-0.3-0.106-0.23.38

Advantage

LossStateValueLoss为什么

天空

蓝色

是

？

光

因为

散射

的

。Prompt+ResponseReward-0.1-0.24-0.72-0.46-0.96-0.9-0.3-0.106-0.23.38

StateValuePredicted-0.91-0.84-0.76-0.8-0.580.30.12.21.23.25ValueHeadLabel：方法一：方差太大方法二：

平衡方差和偏差

Loss

1.进行重要性采样的模型可以与参考模型不一样。2.KL散度已经在Reward中体现，已经通过Reward进入了Los