GAN (v11)Special Structure (v6)李宏毅深度学习

1、Generative Adversarial Network (GAN)Restricted Boltzmann Machine: .tw/tlkagk/courses/MLDS_2015_2/Lecture/RBM%20(v2).ecm.mp4/index.htmlGibbs Sampling:.tw/tlkagk/courses/MLDS_2015_2/Lecture/MRF%20(v2).ecm.mp4/index.htmlOutlook:NIPS 2016 Tutorial: Generative Adversarial NetworksAuthor: Ian GoodfellowPa

2、per: /abs/1701.00160Video: /Events/Neural-Information-Processing-Systems-Conference/Neural-Information-Processing-Systems-Conference-NIPS-2016/Generative-Adversarial-NetworksYou can find tips for training GAN here: /soumith/ganhacksReviewGenerationDrawing?Writing Poems?/index.php?s=/Lot/44547Review:

3、 Auto-encoderAs close as possibleNNEncoderNNDecodercodeNNDecodercodeRandomly generate a vector as codeImage ?Review: Auto-encoderNNDecodercode2D-1.51.5NNDecoderNNDecoderReview: Auto-encoder-1.51.5NNEncoderNNDecodercodeinputoutputAuto-encoderVAENNEncoderinputNNDecoderoutputm1m2m3From a normal distrib

4、utionX+Minimize reconstruction errorexpMinimizeAuto-Encoding Variational Bayes, /abs/1312.6114Problems of VAEIt does not really try to simulate real imagesNNDecodercodeOutputAs close as possibleOne pixel difference from the targetOne pixel difference from the targetRealisticFakeThe evolution of gene

5、rationNNGeneratorv1Discri-minatorv1Real images:NNGeneratorv2Discri-minatorv2NNGeneratorv3Discri-minatorv3Binary ClassifierThe evolution of generationNNGeneratorv1Discri-minatorv1Real images:NNGeneratorv2Discri-minatorv2NNGeneratorv3Discri-minatorv3GAN - DiscriminatorNNGeneratorv1Real images:Discri-m

6、inatorv1image1/0(real or fake)Something like Decoder in VAERandomly sample a vector11110000GAN - GeneratorDiscri-minatorv1NNGeneratorv1Randomly sample a vector0.13Updating the parameters of generator The output be classified as “real” (as close to 1 as possible)Generator + Discriminator = a networkU

7、sing gradient descent to update the parameters in the generator, but fix the discriminator1.0v2GAN 二次元人物頭像鍊成Source of images: /p/24767059DCGAN: /carpedm20/DCGAN-tensorflowGAN 二次元人物頭像鍊成100 roundsGAN 二次元人物頭像鍊成1000 roundsGAN 二次元人物頭像鍊成2000 roundsGAN 二次元人物頭像鍊成5000 roundsGAN 二次元人物頭像鍊成10,000 roundsGAN 二次元人

8、物頭像鍊成20,000 roundsGAN 二次元人物頭像鍊成50,000 roundsBasic Idea of GANMaximum Likelihood EstimationLikelihood of generating the samplesMaximum Likelihood Estimation/generative-models/It is difficult to compute the likelihood.Basic Idea of GANGenerator GG is a function, input z, output x Given a prior distrib

9、ution Pprior(z), a probability distribution PG(x) is defined by function GDiscriminator DD is a function, input x, output scalarEvaluate the “difference” between PG(x) and Pdata(x)There is a function V(G,D). Hard to learn by maximum likelihoodBasic IdeaGiven G, what is the optimal D* maximizingGiven

10、 x, the optimal D* maximizingAssume that D(x) can have any value hereGiven x, the optimal D* maximizingFind D* maximizing: aDbD0 122Jensen-Shannon divergenceIn the end 0 log 2AlgorithmAlgorithmDecrease JS divergence(?)Decrease JS divergence(?)AlgorithmDecrease JS divergence(?)smallerDont update G to

11、o muchIn practice MaximizeMinimize Cross-entropyBinary ClassifierOutput is D(x)Minimize log D(x)If x is a positive exampleIf x is a negative exampleMinimize log(1-D(x)Positive examplesNegative examplesMaximizeMinimize Minimize Cross-entropyBinary ClassifierOutput is f(x)Minimize log f(x)If x is a po

12、sitive exampleIf x is a negative exampleMinimize log(1-f(x)AlgorithmRepeat k timesLearning DLearning GCan only find lower found ofOnly OnceObjective Function for Generatorin Real ImplementationReal implementation: label x from PG as positiveSlow at the beginningDemoThe code used in demo from:/osh/Ke

13、rasGAN/blob/master/MNIST_CNN_GAN_v2.ipynbIssue about Evaluating the DivergenceEvaluating JS divergenceMartin Arjovsky,Lon Bottou, Towards Principled Methods for Training Generative Adversarial Networks, 2017, arXiv preprintEvaluating JS divergenceJS divergence estimated by discriminator telling litt

14、le information/abs/1701.07875Weak GeneratorStrong GeneratorDiscriminatorReason 1. Approximate by sampling10= 0log2Weaken your discriminator?Can weak discriminator compute JS divergence?DiscriminatorReason 2. the nature of data10= 0log2Usually they do not have any overlapEvaluation/post/773890/Better

15、EvaluationBetterNot really better Add NoiseAdd some artificial noise to the inputs of discriminatorMake the labels noisy for the discriminatorDiscriminator cannot perfectly separate real and generated dataNoises decay over timeMode CollapseMode Collapse Data DistributionGenerated DistributionMode Co

16、llapse What we want In reality Flaw in Optimization?Modified from Ian Goodfellows tutorialThis may not be the reason (based on Ian Goodfellows tutorial) So many GANs Modifying the Optimization of GANfGANWGANLeast-square GANLoss Sensitive GANEnergy-based GANBoundary-seeking GANUnroll GANDifferent Str

17、ucture from the Original GANConditional GANSemi-supervised GANInfoGANBiGANCycle GANDisco GANVAE-GANConditional GANMotivationGeneratorScott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee, “Generative Adversarial Text-to-Image Synthesis”, ICML 2016TextImageScott Reed

18、,Zeynep Akata,Santosh Mohan,Samuel Tenka,Bernt Schiele,Honglak Lee, “Learning What and Where to Draw”, NIPS 2016Han Zhang,Tao Xu,Hongsheng Li,Shaoting Zhang,Xiaolei Huang,Xiaogang Wang,Dimitris Metaxas, “StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks”, arXiv prepring, 2016MotivationChallengeNNTextImage(a point, not a distribution)Text: “train”NN outputConditional GANGconditionPrior distributionLearn to approxima