图文并茂的 CNN 介绍ppt课件.ppt

ConvolutionNeuralNetworkCNN,AtutorialKHWong,1,Introduction,VeryPopular:Toolboxes:cuda-convnetandcaffe(userfriendlier)AhighperformanceClassifier(multi-class)SuccessfulinhandwrittenopticalcharacterOCRrecognition,speechrecognition,imagenoiseremovaletc.EasytoimplementationSlowinlearningFastinclassification,2,Overviewofthisnote,Part1:FullyconnectedBackPropagationNeuralNetworks(BPNN)Part1A:feedforwardprocessingPart1A:feedbackwardprocessingPart2:Convolutionneuralnetworks(CNN)Part2A:feedforwardofCNNPart2B:feedbackwardofCNN,3,Part1,FullyConnectedBackPropagation(BP)neuralnet,4,TheoryFullyconnectedBackPropagationNeuralNet(BPNN),Usemanysamplestotraintheweights,soitcanbeusedtoclassifyanunknowninputintodifferentclassesWillexplainHowtouseitaftertraining:forwardpassHowtotrainit:howtotraintheweightsandbiases(usingforwardandbackwardpasses),5,Training,Howtotrainit:howtotraintheweights(W)andbiases(b)(useforward,backwardpasses)InitializeWandbrandomlyIter=1:all_epocks(eachiscalledanepcok)Forwardpassforeachoutputneuron:Usetrainingsamples:Xclass_t:feedforwardtofindy.Err=error_function(y-t)Backwardpass:FindWandbtoreduceErr.Wnew=Wold+W;bnew=bold+b,6,Part1A,ForwardpassofBackPropagationNeuralNet(BPNN)Recall:Forwardpassforeachoutputneuron:-Usetrainingsamples:Xclass_t:feedforwardtofindy.-Err=error_function(y-t),7,FeedforwardofBackPropagationNeuralNet(BPNN),Insideeachneuron:,Outputneurons,Inputs,8,Sigmodfunctionf(u)anditsderivativef(u),9,Asingleneuron,TheneuralnetcanhavemanylayersInbetweenanyneighboring2layers,asetofneuronscanbefound,EachNeuron,10,BPNNForwardpass,Forwardpassistofindoutputwhenaninputisgiven.Forexample:AssumewehaveusedN=60,000imagestotrainanetworktorecognizec=10numerals.Whenanunknownimageisinput,theoutputneuroncorrespondstothecorrectanswerwillgivethehighestoutputlevel.,10outputneuronsfor0,1,2,.,9,Inputimage,11,Thecriteriatotrainanetwork,Isbasedontheoverallerrorfunction,12,StructureofaBPneuralnetwork,Inputlayer,outputlayer,13,Architecture(exercise:writeformulasforA1(i=4)andA2(k=3),Input:P=9x1Indexedbyj,Hiddenlayer=5neurons,indexedbyiW1=9x5b1=5x1,W1(j=1,i=1),W1(j=2,i=1),W1(j=9,i=1),P(j=1)P(j=2)P(j=3):P(j=9),A1(i=1),P(j=1),P(j=2),P(j=9),Neuroni=1Bias=b1(i=1),W2(i=1,k=1),W2(i=2,k=1),W2(i=5,k=1),A2(k=2),A1,A2,A5,Neuronk=1Bias=b2(k=1),W1(j=1,i=1),W1(j=2,i=1),W1(j=9,i=5),W1(j=3,i=4),A1(i=5),A1(i=1),Outputneurons=3neurons,indexedbykW2=5x3b2=3x1,W2(i=5,k=3),W2(i=1,k=1),W2(i=2,k=2),W2(i=2,k=1),A1(i=2),14,Answer(exercise:writevaluesforA1(i=4)andA2(k=3),P=0.76560.73440.96090.99610.91410.90630.09770.09380.0859W1=0.21120.1540-0.0687-0.02890.0720-0.16660.2938-0.0169-0.1127-b1=0.1441%FindA1(i=4)A1_i_is_4=1/(1+exp-(W1*P+b1)=0.49,15,Numericalexamplefortheforwardpath,FeedforwardGivenumbersofx,wbetc,16,Example:asimpleBPNN,Numberofclasses(no.ofoutputneurons)=3Input9pixels:eachinputisa3x3imageTrainingsamples=3foreachclassNumberofhiddenlayers=1Numberofneuronsinthehiddenlayer=5,17,Architectureoftheexample,InputLayer9x1pixels,outputLayer3x1,18,Part1B,BackwardpassofBackPropagationNeuralNet(BPNN),19,feedback,Feedforward,Feedbackward,20,derivation,21,derivation,22,Numericalexampleforthefeedbackpass,23,Procedure,Fromthelastlayer(output),finddt-yFindd,thenfindwofthewholenetworkFinditerative(forward-backforwardpass)togenerateanewsetofW,untildWissmallTakesalongtime,24,Part2ConvolutionNeuralNetworks,Part2AFeedforwardpartofcnnff(),Matlabexample,25,AnexampleopticalcharteredrecognitionOCR,Exampletest_example_CNN.min,26,OverviewofTest_example_CNN.m,ReaddatabasePartI:cnnsetup.mLayer1:inputlayer(donothing)Layer2convolution(conv.)Layer,outputmaps=6,kernelsize=5x5Layer3sub-sample(subs.)Layer,scale=2Layer4conv.Layer,outputmaps=12,kernelsize=5x5Layer5subs.Layer(outputlayer),scale=2Part2:cnntrain.m%trainwedihgtsusing60,000samplescnnff()%CNNfeedforwardcnndb()%CNNfeedbacktotrainweightedinkernelscnnapplygrads()%updateweightscnntest.m%testthesystemusing10000samplesandshowerrorrate,27,Architecture,Eachoutputneuroncorrespondstoacharacter(0,1,2,.,9etc.),Layer1:ImageInput1x28x28,Layer12:6conv.Maps(C)InputMaps=6OutputMaps=6Fan_in=52=25Fan_out=6x52=150,Layer23:6sub-sampleMap(S)InputMaps=6OutputMaps=12,Layer34:12conv.Maps(C)InputMaps=6OutputMaps=12Fan_in=6x52=150Fan_out=12x52=300,Layer45:12sub-sampleMap(S)InputMaps=12OutputMaps=12,Layer1:Oneinput(I),Layer5:12x4x4,10,Kernel=5x5,2x2,Kernel=5x5,Subs,Subs,Layer4:12x8x8,Layer3:6x12x12,Layer2:6x24x24,Conv.,I=inputC=Conv.=convolutionS=Subs=subsampling,Conv.,2x2,28,Cnnff.mconvolutionneuralnetworksfeedforward,ThisisthefeedforwardpartAssumealltheweightsareinitializedorcalculated,weshowhowtogettheoutputfrominputs.,29,Layer12:,Convolutelayer1withdifferentkernels(map_index1=1,2,.,6)andproduce6outputmapsInputs:inputlayer1,a28x28image6differentkernels:k(1),.,k(6),eachkis5x5,KaredendritesofneuronsOutput:6outputmapseach24x24AlgorithmFor(map_index=1:6)layer_2(map_index)=I*k(map_index)validDiscussionValidmeansonlyconsideroverlappedareas,soiflayer1is28x28,kernelis5x5each,eachoutputmapis24x24InMatlabuseconvn(I,k,valid)Example:I=rand(28,28)k=rand(5,5)size(convn(I,k,valid)ans2424,Layer1:ImageInput(i)1x28x28,Layer12:6conv.Maps(C)InputMaps=6OutputMaps=6Fan_in=52=25Fan_out=6x52=150,Layer1:Oneinput(I),Kernel=5x5,2x2,Layer2(c):6x24x24,Conv.*K(6),I=inputC=Conv.=convolutionS=Subs=subsampling,i,j,Map_index=12:6,Conv.*K(1),30,Layer23:,Layer23:6sub-sampleMap(S)InputMaps=6OutputMaps=12,2x2,Subs,Layer3(s):6x12x12,Layer2(c):6x24x24,Sub-samplelayer2tolayer3Inputs:6mapsoflayer2,eachis24x24Output:6mapsoflayer3,eachis12x12AlgorithmFor(map_index=1:6)Foreachinputmap,calculatetheaverageof2x2pixelsandtheresultissavedinoutputmaps.Henceresolutionisreducedfrom24x24to12x12Discussion,Map_index=12:6,31,Layer34:,Layer34:12conv.Maps(C)InputMaps=6OutputMaps=12Fan_in=6x52=150Fan_out=12x52=300,Kernel=5x5,Layer4(c):12x8x8,Layer3L3(s):6x12x12,Conv.layer3withkernelstoproducelayer4Inputs:6mapsoflayer3(L3i=1:6),eachis12x12Kernelset:totally6x12kernels,eachis5x5,i.e.Ki=1:6j=1:12,eachKijis5x512biasj=1:12inthislayer,eachisascalarOutput:12mapsoflayer4(L4j=1:12),eachis8x8Algorithmfor(j=1:12)for(i=1:6)clearz,i.e.z=0;z=z+covn(L3i,kij,valid)%zis8x8L4j=sigm(z+baisj)%L4jis8x8functionX=sigm(P)X=1./(1+exp(-P);EndDiscussionNormalization?,Index=i=1:6,Index=j=1:12,:,net.layersl.aj,32,Layer45,Layer45:12sub-sampleMap(S)InputMaps=12OutputMaps=12,Layer5:12x4x4,10,Subs,Layer4:12x8x8,2x2,Subsamplelayer4tolayer5Inputs:12mapsoflayer4(L4i=1:12),eachis12x8x8Output:12mapsoflayer5(L5j=1:12),eachis4x4AlgorithmSubsampleeach2x2pixelwindowinL4toapixelinL5DiscussionNormalization?,33,Layer5output,Eachoutputneuroncorrespondstoacharacter(0,1,2,.,9etc.)net.om=1:10,Layer45:12sub-sampleMap(S)InputMaps=12OutputMaps=12,Layer5(L5j=1:12:12x4x4=192Totally192pixels,10,Subsamplelayer4tolayer5Inputs:12mapsoflayer5(L5i=1:12),eachis4x4,soL5has192pixelsintotalOutputlayerweights:Net.ffWm=1:10p=1:192,totalnumberofweightsis192Output:10outputneurons(net.om=1:10)AlgorithmForm=1:10%eachoutputneuronclearnet.fvnet.fv=Net.ffWmall192weight.*L5(allcorresponding192pixels)net.om=sign(net.fv+bias)Discussion,:,Totally192weightsforeachoutputneuron,Sameforeachoutputneuron,34,Part2B,Backpropagationpartcnnbp()cnnapplyweight(),35,cnnbp()overview(outputbacktolayer5,Ref:See/wiki/Backpropagation,36,Layer5to4,Expand1x1to2x2,37,Layer4to3,RotatedconvolutionFinddE/dxatlayer3,38,Layer3to2,Expand1x1to2x2,39,Calculategradient,Fromlater2tolayer3Fromlater3tolayer4Net.ffWNet.ffbfound,40,Detailsofcalcgradients,%part%reshapefeaturevectordeltasintooutputmapstyleL4(c)runexpandonlyL3(s)runconv(rot180,fill),founddL2(c)runexpandonly%Part%calcgradientsL2(c)runconv(valid),founddkanddbL3(s)notrunhereL4(c)runconv(valid),founddkanddbDone,foundthesefortheoutputlayerL5:net.dffW=net.od*(net.fv)/size(net.od,2);net.dffb=mean(net.od,2);,41,cnnapplygrads(net,opts),Fortheconvolutionlayers,L2,L4Fromkanddkfindnewk(weights)Frombanddbfindnewb(bias)FortheoutputlayerL5net.ffW=net.ffW-opts.alpha*net.dffW;net.ffb=net.ffb-opts.alpha*net.dffb;opts.alphaistoadjustlearningrate,42,appendix,43,Architecture,Eachoutpu