基于深强化学习的flappybird

SHANGHAIJIAOTONGUNIVERSITYProjectTitle:PlayingtheGameofFlappyBirdwithDeepReinforcementLearningGroupNumber:G-07GroupMembers:WangWenqingGaoXiaoningContents1 Introduction (10)endforEveryCstepsreset=:endforendforExperimentsThissectionwilldescribeouralgorithm’sparameterssettingandtheanalysisofexperimentresults.ParametersSettingsREF_Ref484591565\hFigure6illustratesourCNN’slayerssetting.Theneuralnetworkshas3CNNhiddenlayersfollowedby2fullyconnectedhiddenlayers.Table1showthedetailedparametersofeverylayer.HerewejustuseamaxpoolinginthefirstCNNhiddenlayer.Also,weusetheReLUactivationfunctiontoproducetheneuraloutput.FigureSEQFigure\*ARABIC6:ThelayersettingofCNN:thisCNNhas3convolutionallayersfollowedby2fullyconnectedlayers.Asfortraining,weuseAdamoptimizertoupdatetheCNN’sparameters.TableSEQTable\*ARABIC1:ThedetailedlayerssettingofCNNLayerInputFiltersizeStrideNumfiltersActivationOutputconv180×80×48×8432ReLU20×20×32max_pool20×20×322×2210×10×32conv210×10×324×4264ReLU5×5×64conv35×5×643×3164ReLU5×5×64fc45×5×64512ReLU512fc55122Linear2REF_Ref484591593\hTable1listsalltheparametersettingofDQN.Weuseadecayedrangingfrom0.1to0.001tobalanceexplorationandexploitation.What’smore,REF_Ref484591626\hTable2showsthatthebatchstochasticgradientdescentoptimizerisAdamwithbatchsizeof32.Finally,wealsoallocatealargereplaymemory.TableSEQTable\*ARABIC2:ThetrainingparametersofDQNParametersvalueObservesteps100000Exploresteps3000000Initial_epsilon0.1Final_epsilon0.001Replay_memory50000batchsize32learningrate0.000001FPS30optimizationalgorithmAdamResultsAnalysisWetrainourmodelabout4millionepochs.REF_Ref484591669\hFigure7showstheweightsandbiasesofCNN’sfirsthiddenlayer.Theweightsandbiasesfinallycentralizearound0,withlowvariance,whichdirectlystabilizeCNN’soutputQ-valueandreduceprobabilityofrandomaction.ThestabilityofCNN’sparametersleadstoobtainingoptimalpolicy.FigureSEQFigure\*ARABIC7:Left(right)figureisthehistogramofweights(biases)ofCNN’sfirsthiddenlayerREF_Ref484591680\hFigure8isthecostvalueofDQNduringtraining.Thecostfunctionhasaslowdowntrend,closeto0after3.5millionepochs.ItmeansthatDQNhaslearnedthemostcommonstatesubspaceandwillperformoptimalactionwhencomingacrossknownstate.Inaword,DQNhasobtaineditsbestactionpolicy.FigureSEQFigure\*ARABIC8:DQN’scostfunction:theplotshowsthetrainingprogressofDQN.Wetrainedourmodelabout4millionepochs.Whenplayingflappybird,ifthebirdgetsthroughthepipe,wegiveareward1,ifdead,give-1,otherwise0.1.REF_Ref484591694\hFigure9istheaveragereturnedrewardfromenvironment.Thestabiltiyinfinaltrainingstatemeansthattheagentcanautomaticallychoosethebestaction,andtheenvironmentgivesthebestrewardinturns.Weknowthattheagentandenvironmenthasenterintoafriendlyinteraction,guaranteeingthemaximaltotalreward.FigureSEQFigure\*ARABIC9:Theaveragereturnedrewardfromenvironment.Weaveragethereturnedrewardevery1000epochs.FromthisREF_Ref484591711\hFigure10,thepredictedmaxQ-valuefromCNNconvergesandstabilizesinavalueafterabout100000.ItmeansthatCNNcanaccuratelypredictthequalityofactionsinspecificstate,andwecansteadilyperformactionswithmaxQ-value.TheconvergenceofmaxQ-valuesstatesthatCNNhasexploredthestatespacewidelyandgreatlyapproximatedtheenvironmentwell.FigureSEQFigure\*ARABIC10:TheaveragemaxQ-valueobtainedfromCNN’soutput.WeaveragethemaxQ-valueevery1000epochs.REF_Ref484591726\hFigure11illustratestheDQN’sactionstrategy.IfthepredictedmaxQ-valueissohigh,weareconfidentthatwewillgetthroughthegapwhenperformtheactionwithmaxQ-valuelikeA,C.IfthemaxQ-valueisrelativelylow,andweperformtheaction,wemighthitthepipe,likeB.Inthefinalstateoftraining,themaxQ-valueisdramaticallyhigh,meaningthatweareconfidenttogetthroughthegapsifperformingtheactionswithmaxQ-value.FigureSEQFigure\*ARABIC11:TheleftmostplotshowstheCNN’spredictedmaxQ-valuefora100framessegmentofthegameflappybird.ThethreescreenshotscorrespondtotheframeslabeledbyA,B,andCrespectively.ConclusionWesuccessfullyuseDQNtoplayflappybird,whichcanoutperformhumanbeings.DQNcanautomaticallylearnknowledgefromenvironmentjustusingrawimagetoplaygameswithoutpriorknowledge.ThisfeaturegiveDQNthepowertoplayalmostsimplegames.Moreover,theuseofCNNasafunctionapproximationallowDQNtodealwithlargeenvironmentwhichhasalmostinfinitestatespace.Lastbutnotleast,CNNcanalsogreatlyrepresentfeaturespacewithouthandcraftedfeatureextractionreducingthemassivemanualwork.

ReferencesC.ClarkandA.Storkey.Teachingdeepconvolutionalneuralnetworkstoplaygo.arXivpreprintarXiv:1412.3409,2014.1.AlexKrizhevsky,IlyaSutskever,andGeoffHinton.Imagenetclassificationwithdeepconvolutionalneuralnetworks.InAdvancesinNeuralInformationProcessingSystems25,pages1106–1114,2012.GeorgeE.Dahl,DongYu,LiDeng,andAlexAcero.Context-dependentpre-traineddeepneuralnetworksforlarge-vocabularyspeechrecognition.Audio,Speech,andLanguageProcessing,IEEETransactionson,20(1):30–42,2012,1.RichardSuttonandAndrewBarto.ReinforcementLearning:AnIntroduction.MITPress,1998.BrianSallansandGeoffreyE.Hinton.Reinforcementlearningwithfactoredstatesandactions.JournalofMachineLearningResearch,5:1063–1088,2004.ChristopherJCHWatkinsandPeterDayan.Q-learning.Machinelearning,8(3-4):279–292,1992.HamidMaei,CsabaSzepesv´ari,ShalabhBhatnagar,andRichardS.Sutton.Towardoff-policylearningcontrolwithfunctionapproximation.InProceedingsofthe27thI