台大-李宏毅-B站机器学习视频-课件神经网络与深度学习RL

DeepReinforcementLearning,Scratchingthesurface,DeepReinforcementLearning,ScenarioofReinforcementLearning,Agent,Environment,Observation,Action,Reward,Dontdothat,State,Changetheenvironment,ScenarioofReinforcementLearning,Agent,Environment,Observation,Reward,Thankyou.,Agentlearnstotakeactionstomaximizeexpectedreward.,State,Action,Changetheenvironment,LearningtopalyGo,Environment,Observation,Action,Reward,NextMove,LearningtopalyGo,Environment,Observation,Action,Reward,Ifwin,reward=1,Ifloss,reward=-1,reward=0inmostcases,Agentlearnstotakeactionstomaximizeexpectedreward.,LearningtopalyGo-Supervisedv.s.Reinforcement,Supervised:ReinforcementLearning,Nextmove:“5-5”,Nextmove:“3-3”,Firstmove,manymoves,Win!,AlphaGoissupervisedlearning+reinforcementlearning.,Learningfromteacher,Learningfromexperience,(Twoagentsplaywitheachother.),Learningachat-bot,Sequence-to-sequencelearning,Learningachat-bot-Supervisedv.s.Reinforcement,SupervisedReinforcement,Agent,.,.,Bad,“Hello”,Say“Hi”,“Byebye”,Say“Goodbye”,Learningachat-bot-ReinforcementLearning,Lettwoagentstalktoeachother(sometimesgenerategooddialogue,sometimesbad),Howoldareyou?,Seeyou.,Seeyou.,Seeyou.,Howoldareyou?,Iam16.,Ithoughyouwere12.,Whatmakeyouthinkso?,Learningachat-bot-ReinforcementLearning,Bythisapproach,wecangeneratealotofdialogues.Usesomepre-definedrulestoevaluatethegoodnessofadialogue,Dialogue1,Dialogue2,Dialogue3,Dialogue4,Dialogue5,Dialogue6,Dialogue7,Dialogue8,Machinelearnsfromtheevaluation,DeepReinforcementLearningforDialogueGeneration/pdf/1606.01541v3.pdf,Moreapplications,Interactiveretrieval,Herearewhatyouarelookingfor.,Isitrelatedto“Election”?,user,USPresident,Isee!,Moreprecisely,please.,Trump,Yes.,Wu&Lee,INTERSPEECH16,Moreapplications,FlyingHelicopterSumitChopra,MichaelAuli,WojciechZaremba,“SequenceLevelTrainingwithRecurrentNeuralNetworks”,ICLR,2016,Example:PlayingVideoGame,Widelystudies:Gym:,Machinelearnstoplayvideogamesashumanplayers,Machinelearnstotakeproperactionitself,Whatmachineobservesispixels,Example:PlayingVideoGame,Spaceinvader,fire,Score(reward),Killthealiens,Termination:allthealiensarekilled,oryourspaceshipisdestroyed.,shield,Example:PlayingVideoGame,SpaceinvaderPlayyourself:,Startwithobservation1,Observation2,Observation3,Example:PlayingVideoGame,Action1:“right”,Obtainreward1=0,Action2:“fire”,(killanalien),Obtainreward2=5,Usuallythereissomerandomnessintheenvironment,Startwithobservation1,Observation2,Observation3,Example:PlayingVideoGame,Aftermanyturns,Action,Obtainreward,GameOver(spaceshipdestroyed),Thisisanepisode.,Learntomaximizetheexpectedcumulativerewardperepisode,DifficultiesofReinforcementLearning,RewarddelayInspaceinvader,only“fire”obtainsrewardAlthoughthemovingbefore“fire”isimportantInGoplaying,itmaybebettertosacrificeimmediaterewardtogainmorelong-termrewardAgentsactionsaffectthesubsequentdataitreceivesE.g.Exploration,Outline,Policy-based,Value-based,LearninganActor,LearningaCritic,Actor+Critic,AlphaGo:policy-based+value-based+model-based,VolodymyrMnih,AdriPuigdomnechBadia,MehdiMirza,AlexGraves,TimothyP.Lillicrap,TimHarley,DavidSilver,KorayKavukcuoglu,“AsynchronousMethodsforDeepReinforcementLearning”,ICML,2016,AsynchronousAdvantageActor-Critic(A3C),Tolearndeepreinforcementlearning,Textbook:ReinforcementLearning:AnIntroductionhttps:/webdocs.cs.ualberta.ca/sutton/book/the-book.htmlLecturesofDavidSilverhttp:/www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching.html(10lectures,1:30each),Policy-basedApproach,LearninganActor,MachineLearningLookingforaFunction,Environment,Observation,Action,Reward,Functioninput,Usedtopickthebestfunction,Functionoutput,Action=(Observation),Actor/Policy,ThreeStepsforDeepLearning,DeepLearningissosimple,NeuralNetworkasActor,NeuralnetworkasActor,Inputofneuralnetwork:theobservationofmachinerepresentedasavectororamatrixOutputneuralnetwork:eachactioncorrespondstoaneuroninoutputlayer,NNasactor,pixels,fire,right,left,Probabilityoftakingtheaction,Whatisthebenefitofusingnetworkinsteadoflookuptable?,0.7,0.2,0.1,generalization,ThreeStepsforDeepLearning,DeepLearningissosimple,NeuralNetworkasActor,GoodnessofActor,Review:Supervisedlearning,y1,y2,y10,Loss,“1”,target,Softmax,Ascloseaspossible,Givenasetofparameters,FindthenetworkparametersthatminimizetotallossL,=1,TotalLoss:,TrainingExample,GoodnessofActor,GivenanactorwithnetworkparameterUsetheactortoplaythevideogameStartwithobservation1Machinedecidestotake1Machineobtainsreward1Machineseesobservation2Machinedecidestotake2Machineobtainsreward2Machineseesobservation3MachinedecidestotakeMachineobtainsreward,Totalreward:=1,Evenwiththesameactor,isdifferenteachtime,WedefineastheexpectedvalueofR,evaluatesthegoodnessofanactor,Randomnessintheactorandthegame,END,GoodnessofActor,Anepisodeisconsideredasatrajectory=1,1,1,2,2,2,=1Ifyouuseanactortoplaythegame,eachhasaprobabilitytobesampledTheprobabilitydependsonactorparameter:|,=|,Sumoverallpossibletrajectory,UsetoplaythegameNtimes,obtain1,2,Samplingfrom|Ntimes,1=1,ThreeStepsforDeepLearning,DeepLearningissosimple,NeuralNetworkasActor,GradientAscent,ProblemstatementGradientascentStartwith010+021+1,=max,=1,2,1,=,121,GradientAscent,=?,=|,=|,=|,UsetoplaythegameNtimes,Obtain1,2,1=1|,donothavetobedifferentiable,Itcanevenbeablackbox.,=1,GradientAscent,=1,1,1,2,2,2,|=?,|=,11|1,1,2|1,12|2,2,3|2,2,=1=1|,+1|,Controlbyyouractor,notrelatedtoyouractor,Actor,left,right,fire,0.1,0.2,0.7,=|,=0.7,GradientAscent,=1,1,1,2,2,2,|=1=1|,+1|,|=?,|=1+=1|,+,+1|,|=1|,Ignorethetermsnotrelatedto,GradientAscent,1=1|,|=1|,=1=1=1|,+,=1=1=1|,Ifinmachinetakeswhenseeingin,ispositive,Tuningtoincrease|,isnegative,Tuningtodecrease|,Itisveryimportanttoconsiderthecumulativerewardofthewholetrajectoryinsteadofimmediatereward,Whatifwereplacewith,GradientAscent,1=1|,|=1|,=1=1=1|,+,=1=1=1|,|,|,Whydividedby|,?,hesamplingdata,In13,takeactiona,shasbeenseenin13,15,17,33,In17,takeactionb,In15,takeactionb,In33,takeactionb,R13=2,R17=1,R15=1,R33=1,AddaBaseline,+,1=1=1|,Itispossiblethatisalwayspositive.,Idealcase,Sampling,a,b,c,a,b,c,Itisprobability,a,b,c,Notsampled,a,b,c,Theprobabilityoftheactionsnotsampledwilldecrease.,1=1=1|,Value-basedApproach,LearningaCritic,Critic,Acriticdoesnotdeterminetheaction.Givenanactor,itevaluatesthehowgoodtheactoris,Anactorcanbefoundfromacritic.,e.g.Q-learning,(nottoday),ThreekindsofCritics,AcriticisafunctiondependingontheactoritisevaluatedThefunctionisrepresentedbyaneuralnetworkStatevaluefunctionWhenusingactor,thecumulatedrewardexpectstobeobtainedafterseeingobservation(state)s,s,scalar,islarge,issmaller,ThreekindsofCritics,State-actionvaluefunction,Whenusingactor,thecumulatedrewardexpectstobeobtainedafterseeingobservationsandtakinga,scalar,=,=,=,fordiscreteactiononly,Howtoestimate,Monte-CarlobasedapproachThecriticwatchesplayingthegame,Afterseeing,Untiltheendoftheepisode,thecumulatedrewardis,Afterseeing,Untiltheendoftheepisode,thecumulatedrewardis,Howtoestimate,Temporal-differenceapproach,+=,-,Someapplicationshaveverylongepisodes,sothatdelayingalllearninguntilanepisodesendistooslow.,Howtoestimate,Thecritichasthefollowing8episodes,=0,=0,END,=1,END,=1,END,=1,END,=1,END,=1,END,=1,END,=0,END,Sutton,v2,Example6.4,(Theactionsareignoredhere.),=?,=3/4,0?,3/4?,Monte-Carlo:,Temporal-difference:,=0,+=,3/4,3/4,0,DeepReinforcementLearning,Actor-Critic,1=1=1|,Actor-Critic,+,Evaluatedbycritic,+1,AdvantageFu