neuralnets-2012-001 13_Lecture13 neuralnets-2012-001 13_Lecture13 neuralnets-2012-001 13_Lecture13 01_The_ups_and_downs_of_back_propagation_10_min

NeuralNetworksforMachineLearningLecture13aTheupsanddownsofbackpropagation Abriefhistoryofbackpropagation Thebackpropagationalgorithmforlearningmultiplelayersoffeatureswasinventedseveraltimesinthe70 sand80 s Bryson Ho 1969 linearWerbos 1974 Rumelhartet al in1981Parker 1985 LeCun 1985 Rumelhartet al 1985 Backpropagationclearlyhadgreatpromiseforlearningmultiplelayersofnon linearfeaturedetectors Butbythelate1990 smostseriousresearchersinmachinelearninghadgivenuponit Itwasstillwidelyusedinpsychologicalmodelsandinpracticalapplicationssuchascreditcardfrauddetection Whybackpropagationfailed Thepopularexplanationofwhybackpropagationfailedinthe90 s Itcouldnotmakegooduseofmultiplehiddenlayers exceptinconvolutionalnets Itdidnotworkwellinrecurrentnetworksordeepauto encoders SupportVectorMachinesworkedbetter requiredlessexpertise producedrepeatableresults andhadmuchfanciertheory Therealreasonsitfailed Computerswerethousandsoftimestooslow Labeleddatasetswerehundredsoftimestoosmall Deepnetworksweretoosmallandnotinitializedsensibly Theseissuespreventeditfrombeingsuccessfulfortaskswhereitwouldeventuallybeabigwin Aspectrumofmachinelearningtasks Low dimensionaldata e g lessthan100dimensions Lotsofnoiseinthedata Notmuchstructureinthedata Thestructurecanbecapturedbyafairlysimplemodel Themainproblemisseparatingtruestructurefromnoise Notidealfornon Bayesianneuralnets TrySVMorGP High dimensionaldata e g morethan100dimensions Thenoiseisnotthemainproblem Thereisahugeamountofstructureinthedata butitstoocomplicatedtoberepresentedbyasimplemodel Themainproblemisfiguringoutawaytorepresentthecomplicatedstructuresothatitcanbelearned Letbackpropagationfigureitout TypicalStatistics ArtificialIntelligence WhySupportVectorMachineswereneveragoodbetforArtificialIntelligencetasksthatneedgoodrepresentations View1 SVM sarejustacleverreincarnationofPerceptrons Theyexpandtheinputtoa verylarge layerofnon linearnon adaptivefeatures Theyonlyhaveonelayerofadaptiveweights Theyhaveaveryefficientwayoffittingtheweightsthatcontrolsoverfitting View2 SVM sarejustacleverreincarnationofPerceptrons Theyuseeachinputvectorinthetrainingsettodefineanon adaptive pheature Theglobalmatchbetweenatestinputandthattraininginput Theyhaveacleverwayofsimultaneouslydoingfeatureselectionandfindingweightsontheremainingfeatures HistoricaldocumentfromAT TAdaptiveSystemsResearchDept BellLabs NeuralNetworksforMachineLearningLecture13bBeliefNets Whatiswrongwithback propagation Itrequireslabeledtrainingdata Almostalldataisunlabeled ThelearningtimedoesnotscalewellItisveryslowinnetworkswithmultiplehiddenlayers Why Itcangetstuckinpoorlocaloptima Theseareoftenquitegood butfordeepnetstheyarefarfromoptimal Shouldweretreattomodelsthatallowconvexoptimization Overcomingthelimitationsofback propagationbyusingunsupervisedlearning Keeptheefficiencyandsimplicityofusingagradientmethodforadjustingtheweights butuseitformodelingthestructureofthesensoryinput Adjusttheweightstomaximizetheprobabilitythatagenerativemodelwouldhavegeneratedthesensoryinput Ifyouwanttodocomputervision firstlearncomputergraphics Thelearningobjectiveforagenerativemodel Maximisep x notp y x Whatkindofgenerativemodelshouldwelearn Anenergy basedmodellikeaBoltzmannmachine Acausalmodelmadeofidealizedneurons Ahybridofthetwo ArtificialIntelligenceandProbability ManyancientGreekssupportedSocratesopinionthatdeep inexplicablethoughtscamefromthegods Today sequivalenttothosegodsistheerratic evenprobabilisticneuron Itismorelikelythatincreasedrandomnessofneuralbehavioristheproblemoftheepilepticandthedrunk nottheadvantageofthebrilliant P H Winston ArtificialIntelligence 1977 ThefirstAItextbook Allofthiswillleadtotheoriesofcomputationwhicharemuchlessrigidlyofanall or nonenaturethanpastandpresentformallogic Therearenumerousindicationstomakeusbelievethatthisnewsystemofformallogicwillmoveclosertoanotherdisciplinewhichhasbeenlittlelinkedinthepastwithlogic ThisisthermodynamicsprimarilyintheformitwasreceivedfromBoltzmann JohnvonNeumann TheComputerandtheBrain 1958 unfinishedmanuscript Themarriageofgraphtheoryandprobabilitytheory Inthe1980 stherewasalotofworkinAIthatusedbagsofrulesfortaskssuchasmedicaldiagnosisandexplorationforminerals Forpracticalproblems theyhadtodealwithuncertainty Theymadeupwaysofdoingthisthatdidnotinvolveprobabilities Graphicalmodels Pearl Heckerman Lauritzen andmanyothersshowedthatprobabilitiesworkedbetter Graphsweregoodforrepresentingwhatdependedonwhat Probabilitiesthenhadtobecomputedfornodesofthegraph giventhestatesofothernodes BeliefNets Forsparselyconnected directedacyclicgraphs cleverinferencealgorithmswerediscovered BeliefNets Abeliefnetisadirectedacyclicgraphcomposedofstochasticvariables Wegettoobservesomeofthevariablesandwewouldliketosolvetwoproblems Theinferenceproblem Inferthestatesoftheunobservedvariables Thelearningproblem Adjusttheinteractionsbetweenvariablestomakethenetworkmorelikelytogeneratethetrainingdata stochastichiddencauses visibleeffects GraphicalModelsversusNeuralNetworks Earlygraphicalmodelsusedexpertstodefinethegraphstructureandtheconditionalprobabilities Thegraphsweresparselyconnected Researchersinitiallyfocusedondoingcorrectinference notonlearning Forneuralnets learningwascentral Hand wiringtheknowledgewasnotcool OK maybealittlebit Knowledgecamefromlearningthetrainingdata Neuralnetworksdidnotaimforinterpretabilityorsparseconnectivitytomakeinferenceeasy Nevertheless thereareneuralnetworkversionsofbeliefnets Twotypesofgenerativeneuralnetworkcomposedofstochasticbinaryneurons Energy based WeconnectbinarystochasticneuronsusingsymmetricconnectionstogetaBoltzmannMachine Ifwerestricttheconnectivityinaspecialway itiseasytolearnaBoltzmannmachine Butthenweonlyhaveonehiddenlayer Causal WeconnectbinarystochasticneuronsinadirectedacyclicgraphtogetaSigmoidBeliefNet Neal1992 stochastichiddencauses visibleeffects NeuralNetworksforMachineLearningLecture13cLearningSigmoidBeliefNets LearningSigmoidBeliefNets Itiseasytogenerateanunbiasedexampleattheleafnodes sowecanseewhatkindsofdatathenetworkbelievesin Itishardtoinfertheposteriordistributionoverallpossibleconfigurationsofhiddencauses Itishardtoevengetasamplefromtheposterior Sohowcanwelearnsigmoidbeliefnetsthathavemillionsofparameters stochastichiddencauses visibleeffects Thelearningruleforsigmoidbeliefnets Learningiseasyifwecangetanunbiasedsamplefromtheposteriordistributionoverhiddenstatesgiventheobserveddata Foreachunit maximizethelogprob thatitsbinarystateinthesamplefromtheposteriorwouldbegeneratedbythesampledbinarystatesofitsparents j i Eveniftwohiddencausesareindependentintheprior theycanbecomedependentwhenweobserveaneffectthattheycanbothinfluence Ifwelearnthattherewasanearthquakeitreducestheprobabilitythatthehousejumpedbecauseofatruck truckhitshouse earthquake housejumps 20 20 20 10 p 1 1 0001p 1 0 4999p 0 1 4999p 0 0 0001 posterioroverhiddens Explainingaway JudeaPearl 10 Whyit shardtolearnsigmoidbeliefnetsonelayeratatime TolearnW weneedtosamplefromtheposteriordistributioninthefirsthiddenlayer Problem1 Theposteriorisnotfactorialbecauseof explainingaway Problem2 Theposteriordependsontheprioraswellasthelikelihood SotolearnW weneedtoknowtheweightsinhigherlayers evenifweareonlyapproximatingtheposterior Alltheweightsinteract Problem3 Weneedtointegrateoverallpossibleconfigurationsinthehigherlayerstogetthepriorforfirsthiddenlayer Itshopeless data hiddenvariables hiddenvariables hiddenvariables W prior Somemethodsforlearningdeepbeliefnets MonteCarlomethodscanbeusedtosamplefromtheposterior Neal1992 Butitspainfullyslowforlarge deepbeliefnets Inthe1990 speopledevelopedvariationalmethodsforlearningdeepbeliefnets Theseonlygetapproximatesamplesfromtheposterior Learningwithsamplesfromthewrongdistribution Maximumlikelihoodlearningrequiresunbiasedsamplesfromtheposterior Whathappensifwesamplefromthewrongdistributionbutstillusethemaximumlikelihoodlearningrule Doesthelearningstillworkordoesitdocrazythings NeuralNetworksforMachineLearningLecture13dThewake sleepalgorithm Anapparentlycrazyidea It shardtolearncomplicatedmodelslikeSigmoidBeliefNets Theproblemisthatit shardtoinfertheposteriordistributionoverhiddenconfigurationswhengivenadatavector Itshardeventogetasamplefromtheposterior Crazyidea dotheinferencewrong Maybelearningwillstillwork ThisturnsouttobetrueforSBNs Ateachhiddenlayer weassume wrongly thattheposterioroverhiddenconfigurationsfactorizesintoaproductofdistributionsforeachseparatehiddenunit Factorialdistributions Inafactorialdistribution theprobabilityofawholevectorisjusttheproductoftheprobabilitiesofitsindividualterms AgeneraldistributionoverbinaryvectorsoflengthNhas2 Ndegreesoffreedom actually2 N 1becausetheprobabilitiesmustaddto1 AfactorialdistributiononlyhasNdegreesoffreedom individualprobabilitiesofthreehiddenunitsinalayerprobabilitythatthehiddenunitshavestate1 0 1ifthedistributionisfactorial Thewake sleepalgorithm Hintonet al 1995 Wakephase Userecognitionweightstoperformabottom uppass Trainthegenerativeweightstoreconstructactivitiesineachlayerfromthelayerabove Sleepphase Usegenerativeweightstogeneratesamplesfromthemodel Traintherecognitionwe