深度学习及其应用 课件2

AutoMachineLearningDeepLearninganditsApplicationAutoMLPipelineDataCollectionFeatureEngineeringModelandAlgorithmSelectionHyper-parameterOptimizationQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.2SJTUDeepLearningLecture.Definition

ofAutoMLDefinitionofMachineLearningAcomputerprogramissaidtolearnfromexperienceEwithrespecttosomeclassesoftaskTandperformancemeasurePifitsperformancecanimprovewithEonTmeasuredbyPDefinitionofAutoMLAutoMLattemptstoconstructmachinelearningprograms(specifiedbyE,TandPintheabovedefinition),withouthumanassistanceandwithinlimitedcomputationalbudgetsTargetQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.3SJTUDeepLearningLecture.ThreeGoalsGoodPerformancegoodgeneralizationperformanceacrossvariousinputdataandlearningtaskscanbeachievedNoAssistancefromHumanconfigurationscanbeautomaticallydoneformachinelearningtoolsHighComputationalEfficiencytheprogramcanreturnanreasonableoutputwithinalimitedbudget4SJTUDeepLearningLecture.BasicFrameworkEvaluator:measuretheperformanceofthelearningtoolswithconfigurationsprovidedbytheoptimizerOptimizer:updateorgenerateconfigurationsforlearningtoolsQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.5SJTUDeepLearningLecture.AutoMLTasksAutoMLforDataFeatureengineering(TraditionalMachineLearning)AutoAugmentation(DeepLearning)AutoMLforModelModelSelection(TraditionalMachineLearning)NeuralArchitectureSearch(DeepLearning)AutoMLforHyper-parametersHyper-parameterOptimization6SJTUDeepLearningLecture.FeatureEngineeringGoalperformsomepost-processingonoriginalfeaturestoimprovethelearningperformanceFeature

Enhancement

MethodDimensionalReduction:reducingthenumberofrandomvariablesunderconsiderationbyobtainingasetofprincipalvariables.(E.g.PCA,LDA)FeatureGeneration:constructnewfeaturesfromtheoriginalonesbasedonsomepre-definedoperations.(e.g.StandardNormalization)FeatureEncoding:re-interpretsoriginalfeaturesbasedonsomedictionarieslearnedfromthedata.(e.g.SparseCoding)7SJTUDeepLearningLecture.FeatureEngineeringSearchSpaceHyper-parametersdimensionalreduction:

dimensionofnewfeaturesfeatureencoding:sparsityforSparseCodingTheoperationsforfeaturegeneration8SJTUDeepLearningLecture.AutoAugmentationGoalselectefficientdataaugmentationstrategythathelptoimprovetheaccuracyorspeedupthetrainingSearchSpaceDataaugmentationoperations,suchasrotation,flip,crop,etc..Theprobabilitytoconductthedataaugmentationbeforeeachiteration.9SJTUDeepLearningLecture.ModelSelectionGoalOncefeatureshavebeenobtained,weneedtofindamodeltopredictthelabelsSearchSpaceThechoiceofdifferentmodelsThehyper-parametersforeachmodelsQuanmingY,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.10SJTUDeepLearningLecture.NeuralArchitectureSearchGoalAutomaticallysearchforaneuralarchitectureNetworkarchitecturescanfulfillthelearningpurpose,wherefeatureengineeringandmodelselectionarebothdonebyNASSearchSpaceThetypeofoperationineachlayerTheconnectionoftwolayersThehyper-parametersforeachoperation11SJTUDeepLearningLecture.Hyper-parameterOptimizationGoalAutomaticallysearchforagroupofhyper-parameterconfigurationSearchSpaceLearningrateanditsdecayschedulerWeightdecayOptimizeranditshyper-parametersTrainingepochs12SJTUDeepLearningLecture.OptimizerSelectionForcomplextasks,suchasSVManddeepnetworks,optimizationisnotonlythemainconsumerofcomputationalbudgetsbutalsohasagreatimpactonthelearningperformanceGoalAutomaticallyfindanoptimizationalgorithmsothatefficiencyandperformancecanbebalancedSearchSpaceThehyper-parametersforeachoptimizerL-BFGS:thelengthofstoredgradientSGD:thebatchsize,thelearningrate,andthedecayscheduleQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.13SJTUDeepLearningLecture.SearchStrategyGoalAutomaticallyfindtheconfigurationsinthesearchingspace(includingfeatureengineering,modelselection,andoptimizationalgorithmselection),thatcanachievethebestperformanceontheTaskTMethodSimpleSearchHeuristicSearchModel-basedDerivative-FreeOptimizationReinforcementLearningGradientDescentGreedySearch14SJTUDeepLearningLecture.SimpleSearchGridSearchGridsearchhastoenumerateeverypossibleconfigurationinthesearchspace.DiscretizationisnecessarywhenthesearchspaceiscontinuousRandomSearchrandomlysamplesconfigurationsinthesearchspacerandomsearchcanexploremoreonimportantdimensionsthangridsearchQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.15SJTUDeepLearningLecture.HeuristicSearchPropertyInspiredbybiologicbehaviorsandphenomenon.Suitablefornon-convex,non-smooth,orevennon-continuousproblemsProcedureAteachiteration,anewpopulationisgeneratedbasedonthelastoneThefitness(performances)oftheindividualsareevaluatedCoreIdeahowtoupdatethepopulationQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.16SJTUDeepLearningLecture.HeuristicSearchPSO（Particleswarmoptimization）InspiredbythebehaviorofbiologicalcommunitiesthatexhibitbothindividualandsocialbehaviorAteachiteration,thepopulationisupdatedbymovingtowardsthebestindividualsPSOattendstosearchtheneighborhoodsofthebestsamplesEvolutionaryAlgorithmInspiredbybiologicalevolutionGenerationStep:crossoverandmutation.Withcrossovermainlytoexploit,andmutationmainlytoexplore,thepopulationisexpectedtoevolvetowardsbetterperformance.17SJTUDeepLearningLecture.GreedySearchAnaturalstrategytosolvemulti-stepdecision-makingproblemMotivationSelectlocallyoptimaldecisionateachstepwiththeintentoffindingaglobaloptimumPropertyCannotfindtheglobaloptimumUsuallyfindalocaloptimumwhichapproximatestheglobaloptimuminareasonabletimecost18SJTUDeepLearningLecture.Model-BasedDerivative-FreeOptimizationBayesianOptimization(BO)buildaprobabilisticmodelthatmapstheconfigurationstotheirperformancewithuncertaintyAn

acquisitionfunctionbasedontheprobabilisticmodelisdefinedtobalanceexplorationandexploitationduringsearchAteachiteration,anewsampleisgeneratedbyoptimizingtheacquisitionfunction,andusedtoupdatetheprobabilisticmodelClassification-BasedOptimization(CBO)abinary

classifierislearnedbasedontheprevioussamplestodividethesearchspaceintopositiveandnegativeareasnewsamplesarerandomlygeneratedinthepositiveareawhereitismorelikelytogetbetterconfigurations19SJTUDeepLearningLecture.ReinforcementLearningQuanmingYao,MengshuoWang,HugoJE,etal.Takinghumanoutoflearningapplications:Asurveyonautomatedmachinelearning[J].arXivpreprintarXiv:1810.13306,2018.BeabletosolveproblemswithdelayedfeedbacksThepolicyinRLactsastheoptimizerTheactualperformanceintheenvironmentismeasuredbytheevaluator

20SJTUDeepLearningLecture.GradientbasedMethodFocusingonsomedifferentiablelossfunction,continuoushyper-parameterscanbeoptimizedbygradientdescentComparedwithabovemethods,gradientsofferthemostaccurateinformationwherebetterconfigurationslocates.Forsometraditionalmachinelearningmethods,e.g.,LogisticregressionandSVM,theapproximategradientisproposedtosearchcontinuoushyper-parametersThecomputationofexactgradientsreliesontheconvergenceofmodeltraining21SJTUDeepLearningLecture.EvaluationGoalmeasuretheperformanceofthelearningtoolswithconfigurationsprovidedbytheoptimizerThetradeoffbetweenevaluation’saccuracyandtime,whereDEdenotesdirect.BothtimeandaccuracyaremeasuredrelativelytothatofDE.ThegraylinesindicatevarianceinaccuracyobtainedDirect

Evaluationthemodelparametersarelearnedonthetrainingset,andtheperformanceismeasuredonthevalidationsetafterwards.Directevaluationisoftenaccuratebutexpensive22SJTUDeepLearningLecture.EfficientEvaluationSub-samplinguseasubsetofsamplestotrainthemodelsthelesstrainingdataisused,thefasterandmorenoisywillbetheevaluationEarlyStopInclassicalmachinelearning,earlystopisusedtopreventover-fittinginthecontextofAutoML,itisusuallyusedtocutdownthetrainingtimeforunpromisingconfigurationsItintroducesnoiseandbiastotheestimationassomeconfigurationsSurrogateEvaluatorbuildamodelthatpredictstheperformanceofgivenconfigurations,withexperienceofpastevaluationsSimilaralgorithmsasModel-BasedDerivative-FreeOptimization23SJTUDeepLearningLecture.DeepLearninganditsApplicationConvolutionalneuralnetworksFrankRosenblatt,~1957:

PerceptronTheMarkIPerceptronmachinewasthe

first

implementationoftheperceptron

algorithm.Themachinewasconnectedtoacamerathatused20×20cadmiumsulfidephotocellstoproducea400-pixelimage.recognizedlettersofthealphabetupdate

rule:Abitof

history...25Thisimage

byRockyAcostaislicensedunderCC-BY

3.0Rosenblatt,Frank(1957),ThePerceptron--aperceivingandrecognizingautomaton.Report85-460-1,CornellAeronautical

Laboratory.SJTUDeepLearningLecture.Abitof

history...recognizable

mathIllustrationofRumelhartetal.,1986byLaneMcIntosh,copyrightCS231n

20176Rumelhartetal.,1986:Firsttimeback-propagationbecame

popularRumelhart,DavidE.;Hinton,GeoffreyE.;Williams,RonaldJ.(1986-10-09)."Learningrepresentationsbyback-propagatingerrors".Nature.323(6088):533–536.SJTUDeepLearningLecture.[HintonandSalakhutdinov

2006]Reinvigoratedresearch

inDeep

LearningAbitof

history...IllustrationofHintonandSalakhutdinov2006byLaneMcIntosh,copyrightCS231n

20177GeoffreyEHinton,RuslanRSalakhutdinov(2006)."Reducingthedimensionalityofdatawithneuralnetworks".science.

313(5786)SJTUDeepLearningLecture.Abitof

history:Hubel&

Wiesel,1959RECEPTIVEFIELDSOF

SINGLENEURONES

INTHECAT'SSTRIATE

CORTEX1962RECEPTIVEFIELDS,

BINOCULARINTERACTIONANDFUNCTIONALARCHITECTURE

INTHECAT'SVISUAL

CORTEX1968...9SJTUDeepLearningLecture.Abitof

historyTopographicalmappinginthe

cortex:nearbycellsincortexrepresentnearbyregionsinthevisual

fieldRetinotopyimagescourtesyofJesseGomezintheStanfordVision&PerceptionNeuroscience

Lab.10Human

brainVisualcortexSJTUDeepLearningLecture.Hierarchical

organizationIllustrationofhierarchicalorganizationinearlyvisualpathwaysbyLaneMcIntosh,copyrightCS231n

201711SJTUDeepLearningLecture.Abitof

history:Neocognitron[KunihikoFukushima福岛邦彦

1980]“sandwich”architecture(SCSCSC…)simplecells:modifiableparameterscomplexcells:perform

pooling12SJTUDeepLearningLecture.http://personalpage.flsi.or.jp/fukushima/index-e.htmlInspiredbytheexperimentofHubel&

Wiesel,1959KunihikoFukushima,Neocognitron:ASelf-organizingNeuralNetworkModelforaMechanismofPatternRecognitionUnaffectedbyShiftinPosition,Biol.Cybernetics36,193202(1980)Abitofhistory:12SJTUDeepLearningLecture.Perceptron(1957)Neocognitron(198x)NIN(2013)InceptionV1(2014)InceptionV2/V3(2015)MSRANet(2014)ResNet(2015)ResNetV2(2015)InceptionResNetV2(2016)DenseNet2017…Fast-forwardtotoday:ConvNetsareeverywhereDetectionSegmentation[Farabetetal.,

2012]16SJTUDeepLearningLecture.Convolutionalneuralnetworks19SJTUDeepLearningLecture.Dataaugmentation19SJTUDeepLearningLecture.ImageflippingColortransformationImagecroppingDeepLearninganditsApplicationConvolutionalneuralnetworksConvolutionalneuralnetworks19SJTUDeepLearningLecture.Dataaugmentation19SJTUDeepLearningLecture.ImageflippingColortransformationImagecroppingImageNetcompetition19SJTUDeepLearningLecture.ConvolutionalNeuralNetworks30721FullyConnected

Layer32x32x3image->stretchto3072x

110x

3072weightsactivationinput1101

number:theresultoftakingadotproductbetweenarowofWandthe

input(a3072-dimensionaldot

product)19SJTUDeepLearningLecture.32203ConvolutionLayer32x32x3image->preservespatial

structurewidthheight32depthSJTUDeepLearningLecture.ConvolutionLayer32x32x3

image5x5x3

filter32Convolvethefilterwiththe

imagei.e.“slideovertheimage

spatially,computingdot

products”32321SJTUDeepLearningLecture.ConvolutionLayer32x32x3

image5x5x3

filter32Convolvethefilterwiththe

imagei.e.“slideovertheimage

spatially,computingdot

products”323Filtersalwaysextendthefulldepthoftheinput

volume22SJTUDeepLearningLecture.32332x32x3

image5x5x3

filter321

number:theresultoftakingadotproductbetweenthefilterandasmall5x5x3chunkofthe

image(i.e.5*5*3=75-dimensionaldotproduct+

bias)ConvolutionLayer23SJTUDeepLearningLecture.32332x32x3

image5x5x3

filter32convolve(slide)over

allspatial

locationsactivation

map2412828ConvolutionLayerSJTUDeepLearningLecture.3232332x32x3

image5x5x3

filterconvolve(slide)over

allspatial

locationsactivation

maps12828considerasecond,green

filter25ConvolutionLayerSJTUDeepLearningLecture.323628Forexample,ifwehad65x5filters,we’llget6separateactivation

maps:activation

maps3228Convolution

LayerWestacktheseuptogeta“newimage”ofsize

28x28x6!26ConvolutionLayerSJTUDeepLearningLecture.ConvolutionLayerPreview:ConvNetisasequenceofConvolutionLayers,interspersedwithactivation

functions32323CONV,ReLUe.g.65x5x3filters28286CONV,ReLUe.g.

105x5x6filtersCONV,ReLU27….102424SJTUDeepLearningLecture.57

SJTUDeepLearningLecture.ConvolutionlayerPreview[ZeilerandFergus

2013]ConvolutionLayer28SJTUDeepLearningLecture.PreviewConvolutionLayer29SJTUDeepLearningLecture.example5x5

filters

(32in

total)Wecallthelayerconvolutionalbecauseitisrelatedto

convolutionoftwo

signals:elementwisemultiplicationandsum

ofafilterandthesignal

(image)onefilter

=>oneactivation

mapConvolutionLayer30SJTUDeepLearningLecture.DeepLearninganditsApplicationConvolutionalneuralnetworks下Acloserlookatspatial

dimensions:32332x32x3

image5x5x3

filter32convolve(slide)over

allspatial

locationsactivation

map3212828SpatialdimensionsSJTUDeepLearningLecture.33Acloserlookatspatial

dimensions:77x7input

(spatially)assume3x3

filter7SpatialdimensionsSJTUDeepLearningLecture.Spatialdimensions34Acloserlookatspatial

dimensions:77x7input

(spatially)assume3x3

filter7SJTUDeepLearningLecture.onebyone=>5x5

output357Acloserlookatspatial

dimensions:77x7input

(spatially)assume3x3

filterSpatialdimensionsSJTUDeepLearningLecture.7x7input(spatially)assume3x3filterappliedwithstride

23677Acloserlookatspatial

dimensions:SpatialdimensionsSJTUDeepLearningLecture.7x7input(spatially)assume3x3filterappliedwithstride

23777Acloserlookatspatial

dimensions:SpatialdimensionsSJTUDeepLearningLecture.7x7input(spatially)assume3x3filterappliedwithstride

2=>3x3

output!3877Acloserlookatspatial

dimensions:Spatial

dimensionsSJTUDeepLearningLecture.7x7input(spatially)assume3x3filterappliedwithstride

3?3977Acloserlookatspatial

dimensions:doesn’t

fit!cannotapply3x3filter

on7x7inputwithstride

3.SpatialdimensionsSJTUDeepLearningLecture.NNFF40Output

size:(N-F)/stride+

1e.g.N=7,F=

3:stride1=>(7-3)/1+1=

5stride2=>(7-3)/2+1=

3stride3=>(7-3)/3+1=2.33SpatialdimensionsSJTUDeepLearningLecture.Inpractice:Commontozeropadtheborder410000000000e.g.input

7x73x3filter,appliedwithstride

1padwith1pixelborder=>whatisthe

output?(recall:)(N-F)/stride+

1SJTUDeepLearningLecture.Inpractice:Commontozeropadtheborder420000000000e.g.input

7x73x3filter,appliedwithstride

1padwith1pixelborder=>whatisthe

output?7x7

output!SJTUDeepLearningLecture.•43Inpractice:Commontozeropadthebordere.g.input

7x73x3filter,appliedwithstride

1padwith1pixelborder=>whatisthe

output?7x7

output!ingeneral,commontoseeCONVlayerswithstride1,filtersofsizeFxF,andzero-padding

with(F-1)/2.(willpreservesize

spatially)e.g.

F=3=>z