machine learning2019s课件meta2v按一下以編輯母片標題樣式按一下以編輯母片文字樣式第二層第三層

1、Meta Learning (Part 2):Gradient Descent as LSTMGradient Descent as LSTMHung-yi LeeInit Compute GradientUpdateTrainingDataCompute GradientUpdateTrainingDataNetworkStructureCan we learn more than initialization parameters? The learning algorithm looks like RNN.Recurrent Neural Networkfh0h1y1x1fh2y2x2f

2、h3y3x3No matter how long the input/output sequence is, we only need one function fh and h are vectors with the same dimensionLSTMc change slowlyh change fasterct is ct-1 added by somethinght and ht-1 can be very differentNaveRNNhtytxtht-1LSTMytcththt-1ct-1xtxtzzizfzoht-1ct-1zxtht-1Wzixtht-1Wizfxtht-

3、1Wfzoxtht-1WoReview: LSTMinputforgetoutputhtxtzzizfzoytht-1ct-1cttanhReview: LSTMxtzzizfzoytht-1ct-1ctxt+1zzizfzoyt+1htct+1tanhtanhhtReview: LSTMhtxtzzizfzoytht-1ct-1cttanhSimilar to gradient descent based algorithmall “1”xtzzizfht-1ct-1ctSimilar to gradient descent based algorithmall “1”Dynamic lea

4、rning rateSomething like regularizationOtherLSTM for Gradient Descent “LSTM”“LSTM”“LSTM”3 training stepsTesting DataBatch from trainBatch from trainBatch from trainLearnableTypical LSTMLearn to minimizeIndependence assumptionThe LSTM used only has one cell. Share across all parameters “LSTM”“LSTM”“L

5、STM” Reasonable model size In typical gradient descent, all the parameters use the same update rule Training and testing model architectures can be different.Real Implementation “LSTM”“LSTM”“LSTM”Experimental Resultshttps:/ update depends on not only current gradient, but previous gradients.LSTM for

6、 Gradient Descent (v2) “LSTM”“LSTM”“LSTM”3 training stepsTesting DataBatch from trainBatch from trainBatch from train“LSTM”“LSTM”“LSTM”m can store previous gradientsExperimental Results/abs/1606.04474Meta Learning (Part 3)Hung-yi LeeEven more crazy idea Input: Training data and their

7、 labels Testing data Output: Predicted label of testing data Training Data & Labels catcatdogTesting DataLearning + PredictionFace Verificationhttps:/ your phone by FaceTrainingRegistration (Collecting Training data)In each task:Few-shot LearningTraining TasksTesting TasksTrainTestTrainTestYesTr

8、ainTestNoMeta Learning TrainTestNoYesorNoNetworkNetworkNetworkYesNo?Same approach for Speaker VerificationSiamese NetworkNetworkNoCNNSimilarity CNNshareembeddingembeddingTrainTestscoreLarge score Small score YesNoSameSiamese NetworkNetworkNoCNNSimilarity CNNshareembeddingembeddingTrainTestscoreLarge

9、 score Small score YesNoDifferentSiamese Network- Intuitive Explanation Binary classification problem: “Are they the same?”Training SetTesting Set?samedifferentdifferentNetworkSame or DifferentFace1Face2Siamese Network- Intuitive Explanation Learning embedding for facese.g. learn to ignore the backg

10、roundCNNCNNCNNFar awayAs close as possibleTo learn more What kind of distance should we use? SphereFace: Deep Hypersphere Embedding for Face Recognition Additive Margin Softmax for Face Verification ArcFace: Additive Angular Margin Loss for Deep Face Recognition Triplet loss Deep Metric Learning usi

11、ng Triplet Network FaceNet: A Unified Embedding for Face Recognition and ClusteringN-way Few/One-shot Learning Example: 5-ways 1-shotTraining Data (Each image represents a class)Testing DataNetwork (Learning + Prediction)一花二乃三玖四葉五月三玖Testing DataPrototypical Network/abs/1703.05175CNNCNNCNNCNNCNNCNN= similarityMatching Network/abs/1606.04080Bidirectional LSTMCNN= similarityConsidering the relationship among the training examplesR