扩散模型教程 -Diffusion models

LauraSánchezGarcíaJulioAntonioSotoVicenteIEUniversity(C4__466671-AdvancedArtificialIntelligence)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20241/841Intro2DenoisingDiffusionProbabilisticModels3Advancementsandimprovements4Largediffusionmodels5BeyondimagegenerationIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20242/84IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20243/84A(certainlynotcomplete)list:•LatentVariablemodels(incl.VAEs)•Autoregressivemodels(incl.GPT-styleLanguageModels)•GANs•Flow-basedmodels(incl.NormalizingFlows)•Energy-BasedModels(incl.Score-basedmodels)•Diffusionmodels(kindofmixofallpreviouspoints)•CombinationsIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20244/84ImageImagegenerationSource:Hoelal.[2020]”AphotoofaCorgidogridingabikeinTimesSquare.Itiswearingsunglassesandabeachhat.”Source:Sahariaetal.[2022]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20245/84Source:Brooksetal.[2024]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20246/84IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20247/84Outline•Theforwardprocess•TheNice™property•Thereverseprocess•Lossfunction•Trainingalgorithm•Themodel•SamplingalgorithmIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20248/84IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20249/84”Creatingnoisefromdataiseasy;creatingdatafromnoiseisgenerativemodeling.”Songetal.,2020IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202410/84”Creatingnoisefromdataiseasy;creatingdatafromnoiseisgenerativemodeling.”Songetal.,2020DDPMsprogressivelygenerateimagesoutofnoiseSource:Hoetal.[2020]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202410/84Inorderforamodeltolearnhowtodothat,weneedaprocesstogeneratesuitabletrainingdataintheformof(noise,image)pairsIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202411/84Inorderforamodeltolearnhowtodothat,weneedaprocesstogeneratesuitabletrainingdataintheformof(noise,image)pairsWewillrefertotheprocessofcreatingthetrainingdataastheforwarddiffusionprocess,whichwillprogressivelymakeanimagenoisierIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202411/84Inorderforamodeltolearnhowtodothat,weneedaprocesstogeneratesuitabletrainingdataintheformof(noise,image)pairsWewillrefertotheprocessofcreatingthetrainingdataastheforwarddiffusionprocess,whichwillprogressivelymakeanimagenoisierIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202411/84Ourmodelwilllearnhowtorevertthatnoisingprocess,throughwhatitisknownasthereversediffusionprocess→progressivelydenoisinganoisyimageIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202412/84Ourmodelwilllearnhowtorevertthatnoisingprocess,throughwhatitisknownasthereversediffusionprocess→progressivelydenoisinganoisyimageIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202412/84Ourmodelwilllearnhowtorevertthatnoisingprocess,throughwhatitisknownasthereversediffusionprocess→progressivelydenoisinganoisyimageOncetrained,themodelshouldhavethereforelearnedhowtodenoiseimagesSowecangeneratesomepurelyrandomnoise,runitthroughourmodelandgetbackanimagegeneratedoutofpurenoise!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202412/84Moreformally,DDPMsworkthroughmanystepstwhichare0,1,…,T•x0istheoriginalimage•q(xt∣xt−1)istheforwarddiffusionprocess•pθ(xt−1∣xt)willbethereversediffusionprocess(learnedbyourmodelwithweightsθ)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202413/84Moreformally,DDPMsworkthroughmanystepstwhichare0,1,…,T•x0istheoriginalimage•q(xt∣xt−1)istheforwarddiffusionprocess•pθ(xt−1∣xt)willbethereversediffusionprocess(learnedbyourmodelwithweightsθ)DuringforwarddiffusionweaddGaussian(Normal)noisetotheimageineveryt,producingnoisyimagesx1,x2,…xTAstbecomeshigher,theimagebecomesmoreandmorenoisyIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202413/84q(xt∣xt−1)≔√1−βt⋅xt−1+N(0,βtI)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84q(xt∣xt−1)≔√1−βt⋅xt−1+N(0,βtI)Takeanimageatsomepointt−1GenerateGaussiannoisefromanisotropicmultivariateNormalofsizext,Scalext−1valuesby√1−βt(sodatascaledoesnotgrowasweaddnoise)AddthenoisethethescaledimageIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84q(xt∣xt−1)≔√1−βt⋅xt−1+N(0,βtI)Takeanimageatsomepointt−1GenerateGaussiannoisefromanisotropicmultivariateNormalofsizext,Scalext−1valuesby√1−βt(sodatascaledoesnotgrowasweaddnoise)AddthenoisethethescaledimageItcanbedirectlycomputedasq(xt∣xt−1)≔N(xt;√1−βtxt−1,βtI)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84q(xt∣xt−1)≔√1−βt⋅xt−1+N(0,βtI)Takeanimageatsomepointt−1GenerateGaussiannoisefromanisotropicmultivariateNormalofsizext,Scalext−1valuesby√1−βt(sodatascaledoesnotgrowasweaddnoise)AddthenoisethethescaledimageItcanbedirectlycomputedasq(xt∣xt−1)≔N(xt;√1−βtxt−1,βtI)noiseisaddedineachstept11Inthepaperitismadetogrowlinearlyfromβ1=10−4toβT=0.02forT=1000IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84Thefullforwardprocessistherefore:Tq(x1:T|x0)污Πq(xt|xt—1)t=1ForalargeT,thefinalimageisbasicallyonlynoise(alloriginalimageinfoisessentiallylost),soitbecomesroughlyxT~N(0,I)Demohere!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202415/84Tricktogetanyxtfromx0withouthavingtocomputetheintermediatesteps.Παs.WecanusethereparametrizationtrickfortheNormaldistributiontoget:q(xt|x0)=N(xt;√α- Inthepaperthisisdescribedas”Anotableproperty”.IbelievethatthefirsttocallthisasanicepropertywasWeng[2021].WewillcallittheNice™propertyIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202416/84Tricktogetanyxtfromx0withouthavingtocomputetheintermediatesteps.Παs.WecanusethereparametrizationtrickfortheNormaldistributiontoget:DetailsinAppendixA!q(xt|x0)=N(xt;√α-DetailsinAppendixA!•Easier,fastercomputation•Anyimagestatextcomesfroma(Normal)probabilitydistribution,drasticallysimplyfingderivationsDemohere!Inthepaperthisisdescribedas”Anotableproperty”.IbelievethatthefirsttocallthisasanicepropertywasWeng[2021].WewillcallittheNice™propertyIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202416/84WewilltrainamodelpθtolearntoperformthereverseprocessStartingfromp(xT)=N(0,I),itwilltrytorecreatetheimage!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202417/84WewilltrainamodelpθtolearntoperformthereverseprocessStartingfromp(xT)=N(0,I),itwilltrytorecreatetheimage!pθ(xt1|xt)污N(xt1;μθ(xt,t),Σθ(xt,t))WillbeaneuralnetworkpredictionWillbesettoavalueσIbasedonβtIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202417/84WewilltrainamodelpθtolearntoperformthereverseprocessStartingfromp(xT)=N(0,I),itwilltrytorecreatetheimage!pθ(xt1|xt)污N(xt1;μθ(xt,t),Σθ(xt,t))WillbeaneuralnetworkpredictionWillbesettoavalueσIbasedonβtAndTpθ(x0:T)污p(xT)Πpθ(xt—1|xt)t=1IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202417/84SummarySummaryIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202418/84SummarySummaryTheforwardprocessposterioristheground-truthreversediffusionprocessthatthemodelwilllearntoapproximate!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202418/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:⎡T⎤EqIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:⎡T⎤EqIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:⎡T⎤EqDetailsDetailsinAppendixB!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:⎡T⎤Eq•LT→priormatchingterm.Hasnolearnableparameters,soDetailsinAppendixB!•LtDetailsinAppendixB!•L0→reconstructionterm.Onlylearninghowtogofromx1tox0,soauthorsendedupignoringit(simplerandbetterresults)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84LossthereforefocusesonLt−1:Where:•q(xt−1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0•pθ(xt−1∣xt)willbeourlearnedreverseprocessasseeninslide17 IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossthereforefocusesonLt−1:Where:•q(xt−1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0xt−1∣xt)willbeourlearnedreverseprocessasseeninslide17Theforwardprocessposterioristractableandcanbecomputedas:̃q(xt−1∣xt,x0)=N(xt−1;t(xt,x0),βtI)WhereIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossthereforefocusesonLt−1:Where:•q(xt−1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0xt−1∣xt)willbeourlearnedreverseprocessasseeninslide17Theforwardprocessposterioristractableandcanbecomputedas:̃q(xt−1∣xt,x0)=N(xt−1;t(xt,x0),βtI)WhereDetailsinAppendixC!DetailsinAppendixC!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossthereforefocusesonLt−1:Where:•q(xt−1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0xt−1∣xt)willbeourlearnedreverseprocessasseeninslide17Theforwardprocessposterioristractableandcanbecomputedas:̃q(xt−1∣xt,x0)=N(xt−1;t(xt,x0),βtI)WhereDetailsinAppendixC!DetailsinAppendixC!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt−1∣xt,x0)andthereverseprocessthatourmodelwilllearnIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt−1∣xt,x0)andthereverseprocessthatourmodelwilllearnSincebothareNormaldistributions,thisKLdivergenceis: Eqt(xt,x0)−μθ(xt,t)∥]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt−1∣xt,x0)andthereverseprocessthatourmodelwilllearnpxt−1∣xt)SincebothareNormaldistributions,thisKLdivergenceis: ]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)However:authorsdecideinsteadtopredictthenoiseaddedduringtheforwardprocess.Reformulating: Addednoiseinforwardpass ModelpredictingthenoiseusingxtandtasfeaturesIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt−1∣xt,x0)andthereverseprocessthatourmodelwilllearnpxt−1∣xt)SincebothareNormaldistributions,thisKLdivergenceis: ]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)However:authorsdecideinsteadtopredictthenoiseaddedduringtheforwardprocess.Reformulating: process.Reformulating:Addednoiseinforwardpass ModelpredictingthenoiseusingxtandtasfeaturesDetailsinAppendixD!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt−1∣xt,x0)andthereverseprocessthatourmodelwilllearnpxt−1∣xt)SincebothareNormaldistributions,thisKLdivergenceis: ]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)However:authorsdecideinsteadtopredictthenoiseaddedduringtheforwardprocess.Reformulating: Addednoiseinforwardpass ModelpredictingthenoiseusingxtandtasfeaturesDetailsinAppendixD!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84Source:Hoetal.[2020]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202422/84Source:Hoetal.[2020]Where√tx0+√1+t∈isjustxtcomputedthroughtheNice™property!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202422/84ProposedmodelisaU-Netarchitecture(Ronnebergeretal.[2015])thatincludesself-attentionblocks IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202423/84ProposedmodelisaU-Netarchitecture(Ronnebergeretal.[2015])thatincludesself-attentionblocksTheyalsoincludeGroupNorm(WuandHe[2018])inResNetandself-attentionblockstisaddedoneveryResNetblockthroughpositionalencoding(Vaswanietal.[2017])IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202423/84Oncethemodelistrained,wecangeneratenewimagesby:Source:Hoetal.[2020] IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202424/84Oncethemodelistrained,wecangeneratenewimagesby:Source:Hoetal.[2020]Step4justappliesthereparametrizationtricktothelearnedreverseprocesspθ(xt−1∣xt)=N(xt−1;μθ(xt,t),Σθ(xt,t))IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202424/84Samplingisaniterativeprocess:weprogressivelyremovepredictednoise IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202425/84Samplingisaniterativeprocess:weprogressivelyremovepredictednoiseYoumaywonder:Ifatanysinglestepwearepredictingthefulladdednoise∈,whydon’tweremoveitcompletelyinasinglestep?IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202425/84Samplingisaniterativeprocess:weprogressivelyremovepredictednoiseYoumaywonder:Ifatanysinglestepwearepredictingthefulladdednoise∈,whydon’tweremoveitcompletelyinasinglestep?Answer:DetailsDetailsinAppendixE!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202425/84AdvancementsandimprovementsIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202426/84AdvancementsAdvancementsandimprovementsOutline•Variance/noiseschedulers•Learningthereverseprocessvariance•Fastersampling:DDIMs•Conditionalgeneration•ClassifierGuidance•Classifier-FreeGuidance•Conditioningonimages•ControlNet•ConditioningontextIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202427/84Variance/noiseVariance/noiseschedulersNicholandDhariwal[2021]propot=beingtinywhentiscloseto0(theysetittos=0.008)ComparisonbetweenschedulerinDDPMandNichol&Dhariwal’scosineschedulerproposal.Source:Nichol&Dhariwal[2021]IEUniversity-L.Sánchez,J.A.Soto