与Deep CNNs共享ImageNet分类的细节

SharingdetailofImageNetClassificationwithDeepCNNs林木得OutlineOverviewGoalDatasetModelMotivationArchitectureResultsPartIBasicProblemsActivationFunctionLossFunctionLearningMethodPartIIModelFeaturesReLUNonlinearityTrainingonMultipleGPUsLocalResponseNormalizationOverlappoolingReduceOverfittingDataAugmentationDropoutPartIIIMainphasesPreprocessInitializationStochasticgradientdescentTestReferencesOverviewGoalDatasetModelResultsGoalImageclassificationClassify

theImageNetLSVRC-2010contestimagesinto1000differentclasses.DataSetroughly1.2milliontrainingimages50,000validationimages150,000testingimagesModelMotivation利用自然图像性质

stationarity

of

statistics

locality

of

pixel

independencies模拟神经网络工作机理

receptivefieldModelArchitectureResultsTesterrorinILSVR-2010testsetResultsTesterrorinILSVR-2012testsetsPartIBasicProblemsActivationFunctionLostFunctionLearningMethodActivationFunctionForalllayersexceptoutputlayer: RectifiedLinearUnit(ReLU)TobeconfirmedForoutputlayer:

ReLUandLossFunctionmultinomiallogisticregressionobjective:

tobeconfirmed

LearningMethodGradientDescentTobemorespecific,StochasticGradientDescentwithbatchof128images.PartIIModelFeaturesReLUNonlinearityTrainingonMultipleGPUsLocalResponseNormalizationOverlappoolingReduceOverfittingDataAugmentationDropoutReLUNonlinearityStandardactivationfunction:f(x)=tanh(x)orf(x)=(1+ex)-1

Newinthispaper:

RectifiedLinearUnit(ReLU):

f(x)=max(0,x)

CIFAR-10PerformancecompariseTrainingonMultipleGPUsputshalfofthekernels(orneurons)oneachGPUtheGPUscommunicateonlyincertainlayers.readfromandwritetooneanother’smemorydirectly,Withouthostmachinememoryreducesourtop-1andtop-5errorratesby1.7%and1.2%LocalResponseNormalizationOnvalidationset

k=2,n=5,alpha=10-4,andbeta=0.75

In

realneurons,

横向抑制reducesourtop-1andtop-5errorratesby1.4%and1.2%,respectively.OverlappoolingTraditionally,

non-overlappoolingNewinthispaper:Overlappoolings=2andz=3.educesthetop-1andtop-5errorratesby0.4%and0.3%,respectivelyWhypooling:

1,reducenumberofneuron 2,translateinvarianceOverallarchitectureOverallArchitectureNeuronineachlayers:224x224x3,55x55x96,27x27x256,13x13x394,13x13x394,13x13x256,4096,4096,1000.Almost:650,000neuronsParameterineachlayers:11x11x3x96,5x5x48x256,3x3x256x384,3x3x192x384,3x3x192x256,43264x4096,4096x4096,4096x1000Almost:60millionparametersReduceOverfittingReduceoverfittingisthemostimportantproblemforthismodelDataArgumentationgeneratingimagetranslationsandhorizontalreflec-tions.Train:Afactorof2048moreimagesTest:5x2imagesaveragepredictalteringtheintensitiesoftheRGBchannelsintrainingimages.toeachRGBimagepixelIxy=[IR,IG,IB]Tweaddthefollowingquantity:xyxyxyreducesthetop-1errorratebyover1%.

ReduceOverfittingDropoutMotivation:

Tooexpensivetocombinemanyabovemodelsthattakes5daystotrain

ReduceOverfittingDropoutHOW:

train:settingtozerotheoutputofeachhiddenneuronwithprobability0.5inthefirst2fully-connectlayers.

test:usealltheneuronsbutmultiplytheiroutputsby0.5ReduceOverfittingDropoutCost:

roughlydoublesthenumberofiterationsrequiredtoconverge

PartIIIMainphasesPreprocessInitializationStochasticgradientdescentTestPreprocessdown-sampledtheimagestoafixedresolutionof256x256rescaledtheimagesuchthattheshortersidewasoflength256croppedoutthecentral256x256patchfromtheresultingimagesubtractingthemeanactivityoverthetrainingsetfromeachpixel.Thustrainnetworkonthe(centered)rawRGBvaluesofthepixels.Initializationinitializedtheweightsineachlayerfromazero-meanGaussiandistributionwithstandardde-viation0.01.initializedtheneuronbiasesinthesecond,fourth,andfifth

convolutionallayers,aswellasinthefully-connectedhiddenlayers,withtheconstant1

initializedtheneuronbiasesintheremaininglayerswiththeconstant0learningratewasinitializedat0.01Stochasticgradientdescentwithabatchsizeof128examplesdecayof0.0005Updaterulesdividethelearningrateby10whenthevalidationerrorratestoppedimprovingwiththecurrentlearningrate.learningratereducedthreetimespriortotermination90cyclesthrough1.2millionimages

,took5to6daysTestAttesttime,thenetworkmakesapredictionbyextracting5x2224x224patchesaswellastheirhorizontalreflections(hencetenpatchesinall),andaveragingthepredictionsmadebythenetwork’ssoftmaxlayeronthetenpatches.Attesttime,weusealltheneuronsbutmultiplytheiroutputsby0.5

inthefirsttwofully-connectedlayers.References1,ImageNetClassifi