内容参考讲稿recurrent neural

1、Machine Perception and Interaction Group (MPIG) .cn 跟我学CS231(7)袁洪慧MPIG Open Seminar 0220公众号:mpig_robotRecurrent Neural NetworkRNN的应用RNN的分类Recurrent Neural NetworkRNN的正向传播Truncated Backpropagation LSTMOther RNN Variantssummary目录RNN的应用对于序列化的特征任务，都适合用RNN来解决：情感分析关键字提取语音识别机器翻译股票分析RNN的分类“Vanilla” Neural N

2、etworkVanilla Neural NetworksRecurrent Neural Networks: Process Sequencese.g. Image Captioning image - sequence of wordse.g. Sentiment Classification sequence of words - sentimente.g. Machine Translation seq of words - seq of wordse.g. Video classification on frame levelRecurrent Neural Networkusual

3、ly want to predict a vector at some time stepsRecurrent Neural NetworkhRecurrent Neural NetworkWe can process a sequence of vectors x by applying a recurrence formula at every time step: new statesome function with parameters Wold state input vector atsome time stepRecurrent Neural NetworkWe can pro

4、cess a sequence of vectors x by applying a recurrence formula at every time step:Notice: the same function and the same set of parameters are used at every time step.(Simple) Recurrent Neural NetworkThe state consists of a single “hidden” vector h:RNN的展开图RNN的正向传播RNN: Computational GraphRe-use the sa

5、me weight matrix at every time-step：RNN: Computational Graph: Many to Many RNN: Computational Graph: Many to OneRNN: Computational Graph: One to ManySequence to Sequence: Many-to-one + one-to-manyMany to one: Encode input sequence in a single vectorOne to many: Produce output sequence from single in

6、put vectorTruncated Backpropagation Backpropagation through time梯度截断（Gradient Clipping）为梯度设置阈值，超过该阈值的梯度值都会被cut,这样更新的幅度就不会过大，因此容易收敛。具体做法：Truncated Backpropagation through timeTruncated Backpropagation through timeVanilla RNN Gradient FlowComputing gradient of h0 involves many factors of W (and repeat

7、ed tanh) Bengio et al, “Learning long-term dependencies with gradient descent is difficult”, IEEE Transactions on Neural Networks, 1994 Pascanu et al, “On the difficulty of training recurrent neural networks”, ICML 2013Largest singular value 1: Exploding gradients Largest singular value 1: Vanishing

8、 gradients Gradient clipping: Scale Computing gradient gradient if its norm is too bigSimple-RNN在实际应用中并不多，原因：如果输入越长的话，展开的网络就越深，对于“深度”网络训练的困难最常见的是 Gradient Explode 和 Gradient Vanish 的问题。Simple-RNN基于先前的词预测下一个词，但在一些更加复杂的场景中，例如，“I grew up in France I speak fluent French” “France”则需要更长时间的预测，而随着上下文之间的间隔不断

9、增大时，Simple-RNN会丧失学习到连接如此远的信息的能力。LSTM(Long Short-Term Memory)Long Short Term Memory (LSTM)RNN和LSTM框图 LSTM的核心思想逐步理解 LSTM之遗忘门逐步理解 LSTM之输入门LSTM还需要记住东西，所以有了图示“记忆门”。逐步理解 LSTM逐步理解 LSTM之输出门Other RNN VariantsGRU(Gated Recurrent Unit)GRU是和LSTM功能几乎一样的另一种网络。最终的模型比标准的 LSTM 模型要简单，也是非常流行的变体SummaryRNNs allow a lot

10、of flexibility in architecture design Vanilla RNNs are simple but dont work very well Common to use LSTM or GRU: their additive interactions improve gradient flow Backward flow of gradients in RNN can explode or vanish. Exploding is controlled with gradient clipping. Vanishing is controlled with additive interactions (LSTM) Better/simpler architectures are a hot