11-自动化机器学习

AutoMachineLearningDeepLearninganditsApplicationAutoMLPipelineDataCollectionFeatureEngineeringModelandAlgorithmSelectionHyper-parameterOptimizationQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.2SJTUDeepLearningLecture.Definition

ofAutoMLDefinitionofMachineLearningAcomputerprogramissaidtolearnfromexperienceEwithrespecttosomeclassesoftaskTandperformancemeasurePifitsperformancecanimprovewithEonTmeasuredbyPDefinitionofAutoMLAutoMLattemptstoconstructmachinelearningprograms(specifiedbyE,TandPintheabovedefinition),withouthumanassistanceandwithinlimitedcomputationalbudgetsTargetQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.3SJTUDeepLearningLecture.ThreeGoalsGoodPerformancegoodgeneralizationperformanceacrossvariousinputdataandlearningtaskscanbeachievedNoAssistancefromHumanconfigurationscanbeautomaticallydoneformachinelearningtoolsHighComputationalEfficiencytheprogramcanreturnanreasonableoutputwithinalimitedbudget4SJTUDeepLearningLecture.BasicFrameworkEvaluator:measuretheperformanceofthelearningtoolswithconfigurationsprovidedbytheoptimizerOptimizer:updateorgenerateconfigurationsforlearningtoolsQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.5SJTUDeepLearningLecture.AutoMLTasksAutoMLforDataFeatureengineering(TraditionalMachineLearning)AutoAugmentation(DeepLearning)AutoMLforModelModelSelection(TraditionalMachineLearning)NeuralArchitectureSearch(DeepLearning)AutoMLforHyper-parametersHyper-parameterOptimization6SJTUDeepLearningLecture.FeatureEngineeringGoalperformsomepost-processingonoriginalfeaturestoimprovethelearningperformanceFeature

Enhancement

MethodDimensionalReduction:reducingthenumberofrandomvariablesunderconsiderationbyobtainingasetofprincipalvariables.(E.g.PCA,LDA)FeatureGeneration:constructnewfeaturesfromtheoriginalonesbasedonsomepre-definedoperations.(e.g.StandardNormalization)FeatureEncoding:re-interpretsoriginalfeaturesbasedonsomedictionarieslearnedfromthedata.(e.g.SparseCoding)7SJTUDeepLearningLecture.FeatureEngineeringSearchSpaceHyper-parametersdimensionalreduction:

dimensionofnewfeaturesfeatureencoding:sparsityforSparseCodingTheoperationsforfeaturegeneration8SJTUDeepLearningLecture.AutoAugmentationGoalselectefficientdataaugmentationstrategythathelptoimprovetheaccuracyorspeedupthetrainingSearchSpaceDataaugmentationoperations,suchasrotation,flip,crop,etc..Theprobabilitytoconductthedataaugmentationbeforeeachiteration.9SJTUDeepLearningLecture.ModelSelectionGoalOncefeatureshavebeenobtained,weneedtofindamodeltopredictthelabelsSearchSpaceThechoiceofdifferentmodelsThehyper-parametersforeachmodelsQuanmingY,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.10SJTUDeepLearningLecture.NeuralArchitectureSearchGoalAutomaticallysearchforaneuralarchitectureNetworkarchitecturescanfulfillthelearningpurpose,wherefeatureengineeringandmodelselectionarebothdonebyNASSearchSpaceThetypeofoperationineachlayerTheconnectionoftwolayersThehyper-parametersforeachoperation11SJTUDeepLearningLecture.Hyper-parameterOptimizationGoalAutomaticallysearchforagroupofhyper-parameterconfigurationSearchSpaceLearningrateanditsdecayschedulerWeightdecayOptimizeranditshyper-parametersTrainingepochs12SJTUDeepLearningLecture.OptimizerSelectionForcomplextasks,suchasSVManddeepnetworks,optimizationisnotonlythemainconsumerofcomputationalbudgetsbutalsohasagreatimpactonthelearningperformanceGoalAutomaticallyfindanoptimizationalgorithmsothatefficiencyandperformancecanbebalancedSearchSpaceThehyper-parametersforeachoptimizerL-BFGS:thelengthofstoredgradientSGD:thebatchsize,thelearningrate,andthedecayscheduleQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.13SJTUDeepLearningLecture.SearchStrategyGoalAutomaticallyfindtheconfigurationsinthesearchingspace(includingfeatureengineering,modelselection,andoptimizationalgorithmselection),thatcanachievethebestperformanceontheTaskTMethodSimpleSearchHeuristicSearchModel-basedDerivative-FreeOptimizationReinforcementLearningGradientDescentGreedySearch14SJTUDeepLearningLecture.SimpleSearchGridSearchGridsearchhastoenumerateeverypossibleconfigurationinthesearchspace.DiscretizationisnecessarywhenthesearchspaceiscontinuousRandomSearchrandomlysamplesconfigurationsinthesearchspacerandomsearchcanexploremoreonimportantdimensionsthangridsearchQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.15SJTUDeepLearningLecture.HeuristicSearchPropertyInspiredbybiologicbehaviorsandphenomenon.Suitablefornon-convex,non-smooth,orevennon-continuousproblemsProcedureAteachiteration,anewpopulationisgeneratedbasedonthelastoneThefitness(performances)oftheindividualsareevaluatedCoreIdeahowtoupdatethepopulationQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.16SJTUDeepLearningLecture.HeuristicSearchPSO（Particleswarmoptimization）InspiredbythebehaviorofbiologicalcommunitiesthatexhibitbothindividualandsocialbehaviorAteachiteration,thepopulationisupdatedbymovingtowardsthebestindividualsPSOattendstosearchtheneighborhoodsofthebestsamplesEvolutionaryAlgorithmInspiredbybiologicalevolutionGenerationStep:crossoverandmutation.Withcrossovermainlytoexploit,andmutationmainlytoexplore,thepopulationisexpectedtoevolvetowardsbetterperformance.17SJTUDeepLearningLecture.GreedySearchAnaturalstrategytosolvemulti-stepdecision-makingproblemMotivationSelectlocallyoptimaldecisionateachstepwiththeintentoffindingaglobaloptimumPropertyCannotfindtheglobaloptimumUsuallyfindalocaloptimumwhichapproximatestheglobaloptimuminareasonabletimecost18SJTUDeepLearningLecture.Model-BasedDerivative-FreeOptimizationBayesianOptimization(BO)buildaprobabilisticmodelthatmapstheconfigurationstotheirperformancewithuncertaintyAn

acquisitionfunctionbasedontheprobabilisticmodelisdefinedtobalanceexplorationandexploitationduringsearchAteachiteration,anewsampleisgeneratedbyoptimizingtheacquisitionfunction,andusedtoupdatetheprobabilisticmodelClassification-BasedOptimization(CBO)abinary

classifierislearnedbasedontheprevioussamplestodividethesearchspaceintopositiveandnegativeareasnewsamplesarerandomlygeneratedinthepositiveareawhereitismorelikelytogetbetterconfigurations19SJTUDeepLearningLecture.ReinforcementLearningQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.BeabletosolveproblemswithdelayedfeedbacksThepolicyinRLactsastheoptimizerTheactualperformanceintheenvironmentismeasuredbytheevaluator

20SJTUDeepLearningLecture.GradientbasedMethodFocusingonsomedifferentiablelossfunction,continuoushyper-parameterscanbeoptimizedbygradientdescentComparedwithabovemethods,gradientsofferthemostaccurateinformationwherebetterconfigurationslocates.Forsometraditionalmachinelearningmethods,e.g.,LogisticregressionandSVM,theapproximategradientisproposedtosearchcontinuoushyper-parametersThecomputationofexactgradientsreliesontheconvergenceofmodeltraining21SJTUDeepLearningLecture.EvaluationGoalmeasuretheperformanceofthelearningtoolswithconfigurationsprovidedbytheoptimizerThetradeoffbetweenevaluation’saccuracyandtime,whereDEdenotesd