deep learning techniques and applications深度学习技术与应用

DeepDeepLearning:Techniques DeepChallengetoThetruechallengetoartificialinprovedtobesolvingthetasksthatareeasyforpeopletoperformbuthardforpeopletodescribeformally;solvingtheproblemsthatwesolveintuitively,thatfeelautomatic;likerecognizingspokenwordsorfacesinFlowchartsshowinghowthedifferentpartsofanAIsystemrelatetoeachotherwithindifferentAIdisciplines.Shadedboxesindicatecomponentsthatareabletolearnfromdata.Thefigureshowstwoofthethreehistoricalwavesofartificialneuralnetsresearch,asmeasuredbythefrequencyoftheand“connectionism”or“neuralnetworks”accordingtoDeeplearninghasemoreusefulastheamountofavailabletrainingdatahasKeyTrendsaboutDeepDeeplearningmodelshavegrowninsizeovertimeascomputerhardwareandsoftwareinfrastructurefordeeplearninghasKeyTrendsaboutDeepDeeplearningmodelshavegrowninsizeovertimeascomputerhardwareandsoftwareinfrastructurefordeeplearninghasKeyTrendsaboutDeepDeeplearninghassolvedincreasinglywithincreasingaccuracyovertime.2009

微 的语音识别专家LiDeng和DongYu和Hinton开始合作2011微软的语音识别研究取得成把连续多帧的语音特征并在一起，构 特征逐级地进行信息特征抽取无缝地和传统的语音识别技术相结合，在不引起任何系统额外耗况下大幅度地提升语音识别系统的识别率，LeCun,ConvolutionNeuralNetworks,2012年10月 团队 分类问题上用更深 N取最好结果，使得图像识 2015 公布 错误率DeepDeepLearninginNLP&WhyNeuralThehumancerebralcortexis2to4millimetresinThedifferentcorticallayerseachcontainacharacteristicdistributionneuronalcelltypesandconnectionswithothercorticalandsubcorticalthemoreancientpartofthecerebralcortex,thehippocampus,hasatthreecellularThemostrecentpartofthecerebralcortex,theneocortex(alsocalledisocortex),NeuronsinvariouslayersconnectverticallytoformsmallThecortexisorganizedverticallyincolumnsandhorizontallyinThedifferentregionsofsomatosensoryreceivetheirmaininputsfromdifferentkindsofreceptors.Area3breceivesmostofitsprojectionsfromthesuperficialArea3areceivesinputfromreceptorsinthemuscleInputfromthethalamusarrivesatlayerIV,whereneuronsdistributeinformationupanddownlayers.[Kaasetal.,1979.]Hubel-WieselHubel,DavidH.,andTorstenN.Wiesel."Receptivefieldsofsingleneuronesinthecat'sstriatecortex."TheJournalofphysiology148,no.3(1959):574-591.Hubel-WieselTheNobelPrizeinPhysiologyorMedicine,FurtherBrunoDavidCornell 上不同物UniversityofToronto-MachineLearningGroup(GeoffHinton,RichZemel,RuslanSalakhutdinov,BrendanFrey,RadfordUniversitédeMontréal-LisaLab(YoshuaBengio,PascalVincent,AaronCourville,RolandNewYorkUniversity–YannLecun‘sandRobFergus‘StanfordUniversity–AndrewNg‘sUniversityofOxford–Deeplearninggroup,NandodeFreitasandPhilResearch–JeffDean,SamyBengio,JasonWeston,Marc’AurelioRanzato,DumitruErhan,QuocLeetResearch–LiDengetSUPSI–IDSIA(Jurgen UCBerkeley–BrunoOlshausen‘sUniversityofWashington–PedroDomingos‘IDIAPResearchInstitute-RonanCollobert‘sUniversityofCaliforniaMerced–MiguelA.Carreira-Perpinan‘sUniversityofHelsinki-AapoHyvärinen‘sNeuroinformaticsUniversitédeSherbrooke–Hugo e‘sUniversityofGuelph–GrahamTaylor‘sUniversityofMichigan–HonglakLee‘sTechnicalUniversityofBerlin–Klaus-RobertMuller‘sBaidu–KaiYu‘sAaltoUniversity-JuhaKarhunenandTapaniRaikoU.Amsterdam–MaxWelling‘sU.CaliforniaIrvine–PierreBaldi‘sGhentUniversity–BenjaminShrauwen‘sUniversityofTennessee–ItamarArel‘sIBMResearch–BrianKingsburyetUniversityofBonn–SvenBehnke’sGatsbyUnit@UniversityCollegeLondon–ManeeshSahani,Yee-WhyeTeh,PeterComputationalCognitiveNeuroscienceLab@UniversityofColoradoThere'saninterestinghistoryaboutpeople'schangesintheirattitudestowarddeeparchitecturesandtheshallow2012-Asurveyondeeplearning-onesmallsteptowardHistoricalContextofDeepInaround1960,the1stgenerationofneuralnetworkwasborn(byIt'scapabilityofclassifyingsomebasicshapesliketrianglesandsquaresletpeopleseethepotentialthatarealin ligentmachinewhichcansense,learn,rememberandrecognizelikehuman-beingscanbeinventedwiththistrend.BUT,itsfundamentallimitationssoonbrokepeople'sCriticizingfromMarvinMinsky,OneoftheapparentreasonsisthatthefeaturelayerofthisPerceptronisfixedandcraftedbyhumanbeings,whichisabsolu againstthedefinitionofareal“in Anotherreasonisitssingle-layerstructurelimitsthefunctionsitlearn,e.g,anexclusive-orfunctionisoutofitslearningInaround1985,basedonthePerceptrons,Geoffrey cedtheoriginalsinglefixedfeaturelayerwithseveralhiddenlayers,creatingthe2nd-generationneuralnetwork.viaBack-propagationalgorithm(proposedin1969,practicableinBPdidnotworkwellinThecorrectingsignalwillbeweakenedwhenitpassesbackviamultipleItoftengetstrappedinpoorlocaloptimawhenthebatch-modeorevengradientdescentBPalgorithmisTheseverityincreasessignificantlyasthedepthofthenetworksLearningistooslowacrossmultiplehiddenIn1989,YannLeCunetal.builtadeepneuralnetworkwiththepurposerecognizinghandwrittenZIPcodesonDespitethesuccessofapplyingthealgorithm,thetimetotrainthenetworkondatasetwas y3SVMsloweddownthedevelopmentsofWhenpeopleweretryingtomakeimprovementstoHinton'sneuralnetworkswithrespecttothoseadvantagesTryingtoincreasethetrainingdatasetandestimatingtheinitialweight1993-1995VladimirN.Vapnik,etmadeimprovementsontheoriginalTheycreatinganewfamilySupportVectorSVM——GoodorSVMmakeslearningfastandeasy,duetoitssimpleAppropriatefordatawithsimplestructures,e.g,withasmalloffeaturesorthedatawhichdoesn'tcontainhierarchicalBut,forthedatawhichitselfcontainscomplicatedfeatures,tendstoperformworsebecauseofitssimpleOnewaytosolvethisproblemistoaddapriorknowledgetotheSVMmodelinordertoobtainabetterfeaturelayer.But,it'shardtofindageneralsetofpriorSVM——takesusawayfromtheroadtoareal ligentmachineDespitethefactthatSVMcanworkreallywellinsolvingmanyAIproblems,itisnotagoodtrendtoAIduetoitsfataldeficiency,shallowarchitecture.SVMisstillakindofPerceptronwherethefeaturesareobtainednotlearntfromthedataWiththepurposeoffindinganarchitecturethatmeetstherequirementsabove,someresearchersstartedtolookbacktothemulti-layerneuralnetwork,tryingtoexploititsadvantagesrelatedtodeepandethelimitations...After2010年 国防部DARPA计划首次资助深度学习项目，参与方包括福大学、纽约大学和 2011年，微 和谷歌的语音识别研究人员先后采用DNN技术降低语识别错误率20%-30%，是该领域10年来最大突2012年，Hinton将 分类问题的Top5错误率由26%降至至15%Andrewf建 个音和；2013年，Hinton创立的DNNResearch公司 收购2013年，YannLeCun加 的人工智 After CTR预估（Click-Through-RatePrediction， 检索达到了国际领先水平。2014年，AndrewNg加盟。2013年，腾讯着手建立深度学台Mariana。Mariana面向语音识别、图像识别、推荐等众多应用领域，提供默认算法的并行实现以减少新算DNNGPUDNNCPUBMBM->RBM-BM->RBMBM->RBM-DBN模型被视为由若干个RBM堆叠在一起，可过由低到高逐层训练这些RBM来实现（1）底部RBM以原始输入数据训（2）将底部RBM抽取的特征作为顶部的输入训Hintonin自 ，80年代晚期出主要用于降维，后用于主成分分𝒏输入输出层神经元个m隐藏层神经元个p，q各层神经元的偏w输入层与隐藏层之间的权෥隐藏层与输出层之间的权Y.Bengio,P.Lamblin,D.Popovici,andH. e.Greedylayerwisetrainingofnetworks.InProceedingsofNeuralInformationProcessingSystems(NIPS).2012Hinton–Deep2014/12/9NetworkIn2013ZFVisualizingandUnderstandingConvolutional2014/12/9NetworkIn RNN(RecurrentNeuralallbiologicalneuralnetworksareRNNsimplementdynamicalmathematically,SomeOld