3010神经网络架构搜索-非梯度方法

NeuralArchitectureSearchDeepLearninganditsApplicationNeuralArchitectureSearchGoalAutomaticallysearchforaneuralarchitecturenetworkarchitecturescanfulfillthelearningpurpose,wherefeatureengineeringandmodelselectionarebothdonebyNASNASisitselfalsoatypicalpipelineofAutoMLSearchSpaceThetypeofoperationineachlayerTheconnectionoftwolayersThehyper-parametersforeachoperation2SJTUDeepLearningLecture.PipelineofNAS

iterativeNeuralArchitectureSearchSearchSpaceOperationsforeachlayerConnectionbetweenlayersSearchStrategyReinforcementLearningEvolutionaryAlgorithmDARTSEvaluationMetricDirectEvaluationSurrogateModelProxyDatasetLargeSearchSpaceIntractabletoenumerateallcandidatesInefficientSearchStrategyHardtoinvolveexpertsknowledgeCostlyEvaluationMetricTimeconsumingtoevaluateeachcandidateneuralarchitectureThreeAspectsofNASDifficulties4SJTUDeepLearningLecture.SearchSpaceGlobalSearchSpaceCell-BasedSearchSpace5SJTUDeepLearningLecture.GlobalSearchSpaceChained-structuredSearchSpaceThenetworkisasimplesequencearchitectureThesearchspaceconsistsofoperationineachlayer,andthecorrespondinghyper-parametersChained-structuredSearchSpacewithSkipConnectionsEachlayerreceivesmultipleinputsMartinW,AmbrishR,TejaswiniP.ASurveyonNeuralArchitectureSearch.arXivpreprintarXiv:1905.01392,2019.6SJTUDeepLearningLecture.GlobalSearchSpaceArchitectureTemplateSearchSpaceArchitecturesareseparatedintosequentiallyconnectedsegmentsEachsegmentsisparameterizedbyasetofnodeswithconvolutionsastheiroperation.Segmentsbeginw/aconvolutionandconcludewithmaxpoolingtoreducefeaturedimensions.Themaximumnumberofconvops,andthenumberoffiltersarefixedaspartofthetemplate7Architecturetemplaterelaxestheconnectionpattern,butrestrictsthenumberoffiltersMartinW,AmbrishR,TejaswiniP.ASurveyonNeuralArchitectureSearch.arXivpreprintarXiv:1905.01392,2019.SJTUDeepLearningLecture.Cell-BasedSearchSpaceMotivationEffectivehandcraftedarchitecturesaredesignedw/repetitionsoffixedstructuresCellTwotypes:normalcellandreducedcellReducedcellservestodown-sample,

normalcellmaintainsthespatialsizeTopologyandoperationsinallnormalcellsorreducedcellsarethesameThenumberoffilterscanbedifferentindifferentcells8ElskenT,MetzenJH,HutterF.Neuralarchitecturesearch:Asurvey[J].arXivpreprintarXiv:1808.05377,2018.BasicCellsStackthecellsSJTUDeepLearningLecture.ComparisonGlobalSearchSpaceLargerTime-consumingCell-BasedSearchSpaceSmallerMoreefficientEasytotransferacrossdatasetsArchitecturetemplate(akindofglobalsearchspace)issimilartocell-basedsearchspace.Butitdoesn’trestrictthesametopologyinsegments(cells)9SJTUDeepLearningLecture.SearchStrategyReinforcementLearningEvolutionaryAlgorithmSurrogateModel-BasedOptimizationOne-shotArchitectureSearch10SJTUDeepLearningLecture.NASwithRLRNNasthecontroller(optimizer)toselecttheoperationsineachlayerandthecorrespondinghyper-parametersOnceamodelarchitectureisselectedbythecontroller,itisevaluatedonthevalidationdatasetandreceivesanaccuracyRAccuracyRisthenusedtotrainthecontrollerastherewardZophB,LeQV.Neuralarchitecturesearchwithreinforcementlearning[J].arXivpreprintarXiv:1611.01578,2016.11SJTUDeepLearningLecture.EvolutionaryAlgorithmEvolutionaryalgorithms(EA)arepopulation-basedglobaloptimizerforblack-boxfunctionsEssentialComponentsInitializationParentSelectionRecombinationandMutationSurvivorSelectionProcedureInitializethefirstgenerationofthepopulationSelectparentsfromthepopulationforreproductionApplyrecombinationandmutationoperationstocreatenewindividualsEvaluatethefitnessofthenewindividualsSelectthesurvivorsofthepopulationLoopMartinW,AmbrishR,TejaswiniP.ASurveyonNeuralArchitectureSearch.arXivpreprintarXiv:1905.01392,2019.12SJTUDeepLearningLecture.SurrogateModel-BasedOptimizationSurrogatemodel-basedoptimizersuseasurrogatemodeltoapproximatetheresponsefunctionThesurrogateitselfismodeledasamachinelearningmodelThesurrogatemodelistrainedonameta-datasetwhichcontainsarchitecturedescr