（英文）行业资料Twin Contrastive Learning with Noisy Labels-Huang et al-

1

arXiv:2303.06930v1[cs.CV]13Mar2023

TwinContrastiveLearningwithNoisyLabels

ZhizhongHuang1

HongmingShan2.3*

JunpingZhang1

1ShanghaiKeyLabofIntelligentInformationProcessing,SchoolofComputerScience,FudanUniversity,Shanghai200433,China

2

InstituteofScienceandTechnologyforBrain-inspiredIntelligenceandMOEFrontiersCenterforBrainScience,FudanUniversity,Shanghai200433,China

3ShanghaiCenterforBrainScienceandBrain-inspiredTechnology,Shanghai200031,China(zzhuang19,jpzhang,hmshan}@

Abstract

Learningfromnoisydataisachallengingtaskthatsig-nificantlydegeneratesthemodelperformance.Inthispaper,wepresentTCL,anoveltwincontrastivelearningmodeltolearnrobustrepresentationsandhandlenoisylabelsforclassification.Specifically,weconstructaGaussianmix-turemodel(GMM)overtherepresentationsbyinjectingthesupervisedmodelpredictionsintoGMMtolinklabel-freelatentvariablesinGMMwithlabel-noisyannotations.Then,TCLdetectstheexampleswithwronglabelsastheout-of-distributionexamplesbyanothertwo-componentGMM,takingintoaccountthedatadistribution.Wefurtherproposeacross-supervisionwithanentropyregularizationlossthatbootstrapsthetruetargetsfrommodelpredictionstohandlethenoisylabels.Asaresult,TCLcanlearndiscriminativerepresentationsalignedwithestimatedlabelsthroughmixupandcontrastivelearning.Extensiveexperimentalresultsonseveralstandardbenchmarksandreal-worlddatasetsdemonstratethesuperiorperformanceofTCL.Inparticular,TCLachieves7.5%improvementsonCIFAR-10with90%noisylabel—anextremelynoisyscenario.Thesourcecodeisavailableat

/Hzzone/TCL

.

1.Introduction

Deepneuralnetworkshaveshownexcitingperformanceforclassificationtasks[

13

].Theirsuccesslargelyresultsfromthelarge-scalecurateddatasetswithcleanhumananno-tations,suchasCIFAR-10[

19

]andImageNet[

6

],inwhichtheannotationprocess,however,istediousandcumbersome.Incontrast,onecaneasilyobtaindatasetswithsomenoisyannotations—fromonlineshoppingwebsites[

40

],crowd-sourcing[

42

,

45

],orWikipedia[

32

]—fortrainingaclas-

*Correspondingauthor

sificationneuralnetwork.Unfortunately,themislabelleddataarepronetosignificantlydegradetheperformanceofdeepneuralnetworks.Therefore,thereisconsiderableinter-estintrainingnoise-robustclassificationnetworksinrecentyears[

20

,

21

,

25

,

29

,

31

,

48

].

Tomitigatetheinfluenceofnoisylabels,mostofthemeth-odsinliteratureproposetherobustlossfunctions[

37

,

47

],reducetheweightsofnoisylabels[

35

,

39

],orcorrectthenoisylabels[

20

,

29

,

31

].Inparticular,labelcorrectionmeth-odshaveshowngreatpotentialforbetterperformanceonthedatasetwithahighnoiseratio.Typically,theycorrectthelabelsbyusingthecombinationofnoisylabelsandmodelpredictions[

31

],whichusuallyrequireanessentialitera-tivesampleselectionprocess[

1

,

20

,

21

,

29

].Forexample,Arazoetal.[

1

]usesthesmall-losstricktocarryoutsampleselectionandcorrectlabelsviatheweightedcombination.Inrecentyears,contrastivelearninghasshownpromisingresultsinhandlingnoisylabels[

21

,

21

,

29

].Theyusuallyleveragecontrastivelearningtolearndiscriminativerepre-sentations,andthencleanthelabels[

21

,

29

]orconstructthepositivepairsbyintroducingtheinformationofnearestneighborsintheembeddingspace.However,usingthenear-estneighborsonlyconsidersthelabelnoisewithinasmallneighborhood,whichissub-optimalandcannothandleex-tremelabelnoisescenarios,astheneighboringexamplesmayalsobemislabeledatthesametime.

Toaddressthisissue,thispaperpresentsTCL,anoveltwincontrastivelearningmodelthatexploresthelabel-freeunsupervisedrepresentationsandlabel-noisyannotationsforlearningfromnoisylabels.Specifically,weleveragecontrastivelearningtolearndiscriminativeimagerepresen-tationsinanunsupervisedmannerandconstructaGaus-sianmixturemodel(GMM)overitsrepresentations.Un-likeunsupervisedGMM,TCLlinksthelabel-freeGMMandlabel-noisyannotationsbyreplacingthelatentvariableofGMMwiththemodelpredictionsforupdatingthepa-rametersofGMM.Then,benefittingfromthelearneddata

2

distribution,weproposetoformulatelabelnoisedetectionasanout-of-distribution(OOD)problem,utilizinganothertwo-componentGMMtomodelthesampleswithcleanandwronglabels.ThemeritoftheproposedOODlabelnoisedetectionistotakethefulldatadistributionintoaccount,whichisrobusttotheneighborhoodwithstronglabelnoise.Furthermore,weproposeabootstrapcross-supervisionwithanentropyregulationlosstoreducetheimpactofwronglabels,inwhichthetruelabelsofthesampleswithwrongla-belsareestimatedfromanotherdataaugmentation.Last,tofurtherlearnrobustrepresentations,weleveragecontrastivelearningandMixuptechniquestoinjectthestructuralknowl-edgeofclassesintotheembeddingspace,whichhelpsaligntherepresentationswithestimatedlabels.

Thecontributionsaresummarizedasfollows:

•WepresentTCL,anoveltwincontrastivelearningmodelthatexploresthelabel-freeGMMforunsuper-visedrepresentationsandlabel-noisyannotationsforlearningfromnoisylabels.

•WeproposeanovelOODlabelnoisedetectionmethodbymodelingthedatadistribution,whichexcelsathan-dlingextremelynoisyscenarios.

•Weproposeaneffectivecross-supervision,whichcanbootstrapthetruetargetswithanentropylosstoregu-larizethemodel.

•Experimentalresultsonseveralbenchmarkdatasetsandreal-worlddatasetsdemonstratethatourmethodoutperformstheexistingstate-of-the-artmethodsbyasignificantmargin.Inparticular,weachieve7.5%improvementsinextremelynoisyscenarios.

2.RelatedWork

Contrastivelearning.Contrastivelearningmethods[

3

,

12

,

38

]haveshownpromisingresultsforbothrepresentationlearninganddownstreamtasks.ThepopularlossfunctionisInfoNCEloss[

28

],whichcanpulltogetherthedataaug-mentationsfromthesameexampleandpushawaytheothernegativeexamples.MoCo[

12

]usesamemoryqueuetostoretheconsistentrepresentations.SimCLR[

3

]optimizesInfoNCEwithinmini-batchandhasfoundsomeeffectivetrainingtricks,e.g.,dataaugmentation.However,asunsuper-visedlearning,theymainlyfocusoninducingtransferablerepresentationsforthedownstreamtasksinsteadoftrainingwithnoisyannotations.Althoughsupervisedcontrastivelearning[

17

]canimprovetherepresentationsbyhumanla-bels,itharmstheperformancewhenlabelnoiseexists[

23

].

Learningwithnoisylabels.Mostofthemethodsinlitera-turemitigatethelabelnoisebyrobustlossfunctions[

7

,

25

,

36

,

37

,

41

,

47

],noisetransitionmatrix[

8

,

30

,

35

,

39

],samplese-lection[

11

,

44

],andlabelcorrection[

18

,

20

–

22

,

25

,

29

,

31

,

34

].

Inparticular,labelcorrectionmethodshaveshownpromis-ingresultsthanothermethods.Arazoetal.[

1

]appliedamixturemodeltothelossesofeachsampletodistinguishthenoisyandcleanlabels,inspiredbythefactthatthenoisysampleshaveahigherlossduringtheearlyepochsoftrain-ing.Similarly,DivideMix[

20

]employstwonetworkstoperformthesampleselectionforeachotherandappliesthesemi-supervisedlearningtechniquewherethetargetsarecomputedfromtheaveragepredictionsofdifferentdataaug-mentations.Duetothesuccessofcontrastivelearning,manyattemptshavebeenmadetoimprovetherobustnessofclas-sificationtasksbycombiningtheadvantagesofcontrastivelearning.Zheltonozhskiietal.[

48

]usedcontrastivelearningtopre-traintheclassificationmodel.MOIT[

29

]quantifiesthisagreementbetweenfeaturerepresentationandoriginallabeltoidentifymislabeledsamplesbyutilizingak-nearestneighbor(k-NN)search.RRL[

21

]performslabelclean-ingbytwothresholdsonthesoftlabel,whichiscalculatedfromthepredictionsofpreviousepochsanditsnearestneigh-bors.Sel-CL[

23

]leveragesthenearestneighborstoselectconfidentpairsforsupervisedcontrastivelearning[

17

].

Unlikeexistingmethods[

21

,

23

,

29

]thatdetectthewronglabelswithintheneighborhood,TCLformulatesthewronglabelsastheout-of-distributionexamplesbymodelingthedatadistributionofrepresentationslearnedbycontrastivelearning.Inaddition,weproposeacross-supervisionwithentropyregularizationtobetterestimatethetruelabelsandhandlethenoisylabels.

3.TheProposedTCL

EachimageindatasetD={zi}associateswithanannotationye{1,2,...,K}.Inpractice,someexamplesmaybemislabeled.Weaimtotrainaclassificationnetwork,pθ(y|z)=g(z;θ)eRK,thatisresistanttothenoisylabelsintrainingdata,andgeneralizeswellonthecleantestingdata.Fig.

1

illustratestheframeworkofourproposedTCL.

Overview.Inthecontextofourframework,f(.)andg(.)sharethesamebackboneandhaveadditionalindividualheadstooutputrepresentationsandclasspredictionsfromtworandomandonemixupdataaugmentations.Afterward,therearefourcomponentsinTCL,including(i)modelingthedatadistributionviaaGMMinSec.

3.1

fromthemodelpredictionsandrepresentations;(ii)detectingtheexampleswithwronglabelsasout-of-distributionsamplesinSec.

3.2

;(iii)cross-supervisionbybootstrappingthetruetargetsinSec.

3.3

;and(iv)learningrobustrepresentationsthroughcontrastivelearningandmixupinSec.

3.4

.

3.1.ModelingDataDistribution

GiventheimagedatasetconsistingofNimages,weopttomodelthedistributionofzoveritsrepresentationg=f(z)viaasphericalGaussianmixturemodel(GMM).Afterin-

3

ClassPredictions

2ndview

InputAugmentationBackboneHead

1stview

mixupview

Sec3.3:

Sec3.1:ModelingDataDistribution

CrossSupervision

Crong

?lbWnorwronglWabls?

?lbWn

Sec3.2:LabelNoiseDetection

mixupview

Sec.3.4:

LearningRobustRepresentations

Representations

GMMofDataDistribution

1stview

Encoder

2ndview

MLP

MLP

Figure1.IllustrationoftheproposedTCL.Thenetworksgandfwithsharedencoderandindependenttwo-layerMLPoutputtheclasspredictionsandrepresentations.Then,TCLmodelsthedatadistributionviaaGMM,anddetectstheexampleswithwronglabelsasout-of-distributionexamples.TooptimizeTCL,theseresultsleadtocross-supervisionandrobustrepresentationlearning.

troducingdiscretelatentvariablesze{1,2,...,K}thatdeterminetheassignmentofobservationstomixturecompo-nents,theunsupervisedGMMcanbedefinedas

K

p(U)=Lk=1p(U,z=k)

=Lk=1p(z=k)N(U|uk,σk).(1)

K

whereukisthemeanandσkascalardeviation.Ifweassumethatthelatentvariableszareuniformdistributed,thatis,p(z=k)=1/K,wecandefinetheposteriorprobabilitythatassignszitok-thcluster:

γik=p(zi=k|zi)xN(zi|uk,σk).(2)

Inanidealscenariowhereallthesampleshavecleanlabelsye{1,2,...,K},thediscretelatentvariableszwouldbeidenticaltotheannotationy,andtheparametersuk,σkandlatentvariablezcanbesolvedthroughastandardExpectation-Maximization(EM)algorithm[

5

].

However,inpractice,thelabelsareoftennoisyandthelatentvariablez,estimatedinanunsupervisedmanner,hasnothingtodowiththelabely.Therefore,weareinterestedinconnectinglatentvariablez,estimatedinanunsuper-visedfashion(i.e.label-free),andtheavailableannotationsy,label-noisy,forthetaskoflearningfromnoisylabels.

Tolinkthemtogether,weproposetoinjectthemodelpre-dictionspθ(yi=k|zi),learnedfromnoisylabels,intothelatentvariablesz.Specifically,weproposetoreplacetheun-supervisedassignmentp(zi=k|zi)withnoisy-supervisedassignmentpθ(yi=k|zi).Asaresult,wecanconnectthelatentvariablezwiththelabely,andthususethenoisysupervisiontoguidetheupdateoftheparametersofGMM.

Inparticular,theupdateoftheGMMparametersbecomes

uk=norm╱、,

←ipθ(yi=k|zi),

σk=←ipθ(yi=k|zi)(Ui-uk)(Ui-uk)T

(3)

(4)

wherenorm(.)ise2-normalizationsuchthat|uk|2=1.

3.2.Out-Of-DistributionLabelNoiseDetection

Previousworks[

21

,

23

,

29

]typicallydetectthewronglabelswithintheneighborhood,thatis,usingtheinformationfromnearestneighbors.Itislimitedastheneighboringexamplesareusuallymislabeledatthesametime.Toaddressthisissue,weproposetoformulatelabelnoisedetectionastodetecttheout-of-distributionexamples.

Afterbuildingtheconnectionbetweenthelatentvariableszandlabelsy,weareabletodetectthesamplewithwronglabelsthroughtheposteriorprobabilityinEq.(

2

).Weim-plementitasanormalizedversiontotakeintoaccounttheintra-clusterdistance,whichallowsfordetectingthesampleswithlikelywronglabels:

←kexp(-(Ui-uk)T(Ui-uk)/2σk).

γik=exp╱-(Ui-uk)T(Ui-µk)/2σk、(5)

Sincee2-normalizationhasbeenappliedtobothembeddingsUandtheclustercentersuk,yielding(U-uk)T(U-uk)=2-2UTuk.Therefore,wecanre-writeEq.(

5

)as:

γik=p(zi=k|zi)

=exp(Uuk/σk)\Lkexp(Uuk/σk).(

6)

OncebuilttheGMMoverthedistributionofrepresenta-tions,weproposetoformulatetheconventionalnoisylabel

4

detectionproblemasout-of-distributionsampledetectionproblem.Ourideaisthatthesampleswithcleanlabelsshouldhavethesameclusterindicesafterlinkingtheclusterindexandclasslabel.Specifically,givenoneparticularclassy=k,thesampleswithinthisclasscanbedividedintotwotypes:in-distributionsampleswithcleanlabels,andout-of-distributionsampleswithwronglabels.Therefore,wedefinethefollowingconditionalprobabilitytomeasuretheprobabilityofonesamplewithcleanlabel:

γy=Xìi=p(yi=zi|zi)

=exp(guXi/σXi)′Lkexp(guk/σk).(

7)

AlthoughEqs.(

6

)and(

7

)sharesimilarcalculations,theyhavedifferentmeanings.Eq.(

6

)calculatestheprobabilityofoneexamplebelongingtok-thclusterwhileEq.(

7

)theprobabilityofoneexamplehavingcleanlabel—thatis,yi=zi.Therefore,theprobabilityofoneexamplehavingthewronglabelcanbewrittenasγyXìi=p(yizi|zi)=1-p(yi=zi|zi).

Furthermore,insteadofsettingahuman-tunedthresholdforγy=Xìi,weopttoemployanothertwo-componentGMMfollowing[

1

,

20

]toautomaticallyestimatethecleanproba-bilityγy=Xìiforeachexample.SimilartothedefinitionofGMMinEq.(

1

),thistwo-componentsGMMisdefinedasfollows:

11

p(γy=Xìi)=Lp(γy=Xìi,c)=Lp(c)p(γy=Xìi|c),(8)

c=0c=0

wherecisthenewintroducedlatentvariable:c=1indicatestheclusterofcleanlabelswithhighermeanvalueandviceversusc=0.AftermodelingtheGMMovertheprobabilityofoneexamplehavingcleanlabels,γy=Xìi,weareabletoinfertheposteriorprobabilityofoneexamplehavingcleanlabelsthroughthetwo-componentGMM.

3.3.Cross-supervisionwithEntropyRegularization

Afterthelabelnoisedetection,thenextimportantstepistoestimatethetruetargetsbycorrectingthewrongla-beltoreduceitsimpact,calledlabelcorrection.Previousworksusuallyperformlabelcorrectionusingthetemporalensembling[

25

]orfromthemodelpredictions[

1

,

20

]beforemixupaugmentationwithoutback-propagation.

TCLleveragesasimilarideatobootstrapthetargetsthroughtheconvexcombinationofitsnoisylabelsandthepredictionsfromthemodelitself:

,(9)

whereg(z)andg(z)arethepredictionsoftwoaug-mentations,yithenoisyone-hotlabel,andwie[0,1]rep-resentstheposteriorprobabilityasp(c=1|γy=Xìi)fromthe

two-componentGMMdefinedinEq.(

8

).Whencomput-ingEq.(

9

),westopthegradientfromgtoavoidthemodelpredictionscollapsedintoaconstant,inspiredby[

4

,

10

].

Guidedbythecorrectedlabelsti,weswaptwoaugmen-tationstocomputetheclassificationlosstwice,leadingto

thebootstrapcrosssupervision,formulatedas:

ccross=e╱g(z),t、+e╱g(z),t、,(

10)

whereeisthecross-entropyloss.Thislossmakesthepre-dictionsofthemodelfromtwodataaugmentationsclosetocorrectedlabelsfromeachother.Inasense,ifwi=0,themodelisencouragedforconsistentclasspredictionsbe-tweendifferentdataaugmentations,otherwisewi=1itissupervisedbythecleanlabels.

Inaddition,weleverageanadditionalentropyregulariza-tionlossonthepredictionswithinamini-batchB:

creg=-H╱g(z)\+H(g(z)),(11)

whereH(.)istheentropyofpredictions[

33

].Thefirsttermcanavoidthepredictionscollapsingintoasingleclassbymaximizingtheentropyofaveragepredictions.Thesecondtermistheminimumentropyregularizationtoencouragethemodeltohavehighconfidenceforpredictions,whichwaspreviouslystudiedinsemi-supervisedlearningliterature[

9

]. Althoughbothusingthemodelpredictions,wewouldemphasizethatthecross-supervisioninTCLisdifferent

to[

1

,

20

,

25

]inthreeaspects:(i)bothzandzarein-volvedinback-propagation;(ii)thestrongaugmentation[

3

]usedtoestimatethetruetargetscanpreventtheoverfitting

ofestimatedtargets;and(iii)TCLemploystwoentropy

regularizationtermstoavoidthemodelcollapsetooneclass.Thefinalclassificationlossisgivenasfollows:

ccls=ccross+creg.(12)

3.4.LearningRobustRepresentations

Tomodelthedatadistributionthatisrobusttonoisylabels,weleveragecontrastivelearningtolearntherepre-sentationsofimages.Specifically,contrastivelearningper-formsinstance-wisediscrimination[

38

]usingtheInfoNCEloss[

28

]toenforcethemodeloutputtingsimilarembed-dingsfortheimageswithsemanticpreservingperturbations.Formally,thecontrastivelossisdefinedasfollows:

(13)

←℃炷éexp╱f(z(1))Tf(z)/τ、,

exp╱f(z(1))Tf(z(2))/τ、cctr=-log

whereτisthetemperatureandsistheBexceptz(1).z(1)andz(2)aretwoaugmentationsofz.Intuitively,InfoNCElossaimstopulltogetherthepositivepair(z(1),z(2))from

5

twodifferentaugmentationsofthesameinstance,andpushthemawayfromnegativeexamplesofotherinstances.Con-sequently,itcanencouragediscriminativerepresentationsinapureunsupervised,orlabel-freemanner.

Althoughbeneficialinmodelinglatentrepresentations,

contrastivelearningcannotintroducecompactclasseswith-outusingthetruelabels.Sincethelabelyisnoisy,welever-ageMixup[

46

]toimprovewithin-classcompactness,whichhasbeenshownitseffectivenessagainstlabelnoiseinlitera-ture[

1

,

20

].Specifically,amixuptrainingpair(z,)islinearlyinterpolatedbetween(zi,i)and(zj,j)underacontrolcoefficientλ~Beta(α,α):

,(14)

wherezjisrandomlyselectedwithinamini-batch,andi=(t+t)/2istheaverageofestimatedtruelabelsoftwodataaugmentations.Intuitively,wecaninjectthestructuralknowledgeofclassesintotheembeddingspace

learnedbycontrastivelearning.Thislosscanbewrittenas:

calign=e╱g(z),、+e(p(:|z),),(15)

wherethesecondtermcanaligntherepresentationswithestimatedlabels.Inasense,calignregularizesclassificationnetworkgandencouragesftolearncompactandwell-separatedrepresentations.Furthermore,wewouldpointouttwodifferencesbetweenTCLand[

21

],althoughbothusingmixuptoboosttherepresentations.First,[

21

]doesnotexplicitlymodelthedatadistributionp(:|z)likeTCL.

Second,TCLhasleveragedthefulltrainingdatasetviathe

correctedlabelinsteadofasubsetofcleanexamplesin[

21

],whichleadstostrongerrobustnessofTCLover[

21

]onextremehighlabelnoiseratios.

3.5.Trainingandinference

Theoveralltrainingobjectiveistominimizethesumofalllosses:

c=ccls+cctr+calign.(16)

Wefindthatasimplesummationofalllossesworkswellforalldatasetsandnoiselevels,whichindicatesthestronggeneralizationoftheproposedmethod.Duringinference,thedataaugmentationsaredisabledandtheclasspredictionsareobtainedbyargmaxkpθ(k|女).

ThetrainingalgorithmoftheproposedmethodisshowninAlg.

1

.Inasense,thearchitectureofourmethodleadstoanEM-likealgorithm:(1)theE-stepupdates{(uk,σk)}forTCL,and{wi}foreachsampleinDtoformthetruetargetswiththepredictionsfromanotherdataaugmentations,and(2)theM-stepoptimizesthemodel

Algorithm1:TrainingAlgorithm

Input:DatasetD={(zi,yi)};functions{f,g}Output:Classificationnetworkg.

repeat

E-step:update{(uk,σk)}forTCL,and{wi}foreachsampleinD

M-step:repeat

Randomlysampleamini-batchBfromDforeachziinBdo

Randomlysampletwoaugmentations

c-Eq.(

16

)

andamixupone:{z,z,z}

end

UpdatefandgwithSGDoptimizer.untilanepochfinished;

untilreachingmaxepochs;

parametersbyEq.(

16

)tobetterfitthoseestimatedtargets.Therefore,theconvergenceofTCLcanbetheoreticallyguar-anteed,followingthestandardEMalgorithm.

4.Experiments

Inthissection,weconductexperimentsonmultiplebenchmarkdatasetswithsimulatedandreal-worldlabelnoises.Westrictlyfollowtheexperimentalsettingsinprevi-ousliterature[

20

,

21

,

25

,

29

]forfaircomparisons.

4.1.Experimentsonsimulateddatasets

Datasets.Following[

20

,

21

,

25

,

29

],wevalidateourmethodonCIFAR-10/100[

19

],whichcontains50Kand10Kim-ageswithsize32×32fortrainingandtesting,respectively.Weleave5Kimagesfromthetrainingsetasthevalidationsetforhyperparametertuning,thentrainthemodelonthefulltrainingsetforfaircomparisons.Twotypesoflabelnoisearesimulated:symmetricandasymmetriclabelnoise.Symmetricnoiserandomlyassignsthelabelsofthetrain-ingsettorandomlabelswithpredefinedpercentages,a.k.a,noiseratio,whichincludes20%,50%,80%,and90%ontwodatasetsinthispaper.Asymmetricnoisetakesintoac-counttheclasssemanticinformation,andthelabelsareonlychangedtosimilarclasses(e.g.,truck→automobile).Here,onlyexperimentsontheCIFAR-10datasetwith40%noiseratioforasymmetricnoiseareconducted;otherwise,theclasseswithabove50%labelnoisecannotbedistinguished.

Trainingdetails.Sameaspreviousworks[

20

,

21

,

25

,

29

],weuseaPreActResNet-18[

14

]astheencoder.WeadoptSGDoptimizertotrainourmodelwithamomentumof0.9,aweightdecayof0.001,andabatchsizeof256for200epochs.Thelearningratearelin