人工神经网络原理 英文版

1、FundamentalsofArtificialNeuralNetworksbyMohamadH.Hassoun(MITPress,1995) HYPERLINK / /PrefaceMypurposeinwritingthisbookhasbeentogiveasystematicaccountofmajorconceptsandmethodologiesofartificialneuralnetworksandtopresentaunifiedframeworkthatmakesthesubjectmoreaccessibletostudentsandpractitioners.Thisb

2、ookemphasizesfundamentaltheoreticalaspectsofthecomputationalcapabilitiesandlearningabilitiesofartificialneuralnetworks.Itintegratesimportanttheoreticalresultsonartificialneuralnetworksandusesthemtoexplainawiderangeofexistingempiricalobservationsandcommonlyusedheuristics.Themainaudienceisfirst-yeargr

3、aduatestudentsinelectricalengineering,computerengineering,andcomputerscience.Thisbookmaybeadaptedforuseasaseniorundergraduatetextbookbyselectivechoiceoftopics.Alternatively,itmayalsobeusedasavaluableresourceforpracticingengineers,computerscientists,andothersinvolvedinresearchinartificialneuralnetwor

4、ks.Thisbookhasevolvedfromlecturenotesoftwocoursesonartificialneuralnetworks,asenior-levelcourseandagraduate-levelcourse,whichIhavetaughtduringthelast6yearsintheDepartmentofElectricalandComputerEngineeringatWayneStateUniversity.Thebackgroundmaterialneededtounderstandthisbookisgeneralknowledgeofsomeba

5、sictopicsinmathematics,suchasprobabilityandstatistics,differentialequationsandlinearalgebra,andsomethingaboutmultivariatecalculus.Thereaderisalsoassumedtohaveenoughfamiliaritywiththeconceptofasystemandthenotionofstate,aswellaswiththebasicelementsofBooleanalgebraandswitchingtheory.Therequiredtechnica

6、lmaturityisthatofaseniorundergraduateinelectricalengineering,computerengineering,orcomputerscience.Artificialneuralnetworksareviewedhereasparallelcomputationalmodels,withvaryingdegreesofcomplexity,comprisedofdenselyinterconnectedadaptiveprocessingunits.Thesenetworksarefine-grainedparallelimplementat

7、ionsofnonlinearstaticordynamicsystems.Averyimportantfeatureofthesenetworksistheiradaptivenature,wherelearningbyexamplereplacestraditionalprogramminginsolvingproblems.Thisfeaturemakessuchcomputationalmodelsveryappealinginapplicationdomainswhereonehaslittleorincompleteunderstandingoftheproblemtobesolv

8、edbutwheretrainingdataisreadilyavailable.Anotherkeyfeatureistheintrinsicparallelismthatallowsforfastcomputationsofsolutionswhenthesenetworksareimplementedonparalleldigitalcomputersor,ultimately,whenimplementedincustomizedhardware.Artificialneuralnetworksareviablecomputationalmodelsforawidevarieyofpr

9、oblems,includingpatternclassification,speechsynthesisandrecognition,adaptiveinterfacesbetweenhumansandcomplexphysicalsystems,functionapproximation,imagedatacompression,associativememory,clustering,forecastingandprediction,combinatorialoptimization,nonlinearsystemmodeling,andcontrol.Thesenetworksaren

10、euralinthesensethattheymayhavebeeninspiredbyneuroscience,butnotbecausetheyarefaithfulmodelsofbiologicneuralorcognitivephenomena.Infact,themajorityofthenetworkmodelscoveredinthisbookaremorecloselyrelatedtotraditionalmathematicaland/orstatisticalmodelssuchasoptimizationalgorithms,nonparametricpatternc

11、lassifiers,clusteringalgorithms,linearandnonlinearfilters,andstatisticalregressionmodelsthantheyaretoneurobiologicmodels.Thetheoriesandtechniquesofartificialneuralnetworksoutlinedherearefairlymathematical,althoughthelevelofmathematicalrigorisrelativelylow.InmyexpositionIhaveusedmathematicstoprovidei

12、nsightandunderstandingratherthantoestablishrigorousmathematicalfoundations.Theselectionandtreatmentofmaterialreflectmybackgroundasanelectricalandcomputerengineer.Theoperationofartificialneuralnetworksisviewedasthatofnonlinearsystems:Staticnetworksareviewedasmappingorstaticinput/outputsystems,andrecu

13、rrentnetworksareviewedasdynamicalsystemswithevolvingstate.Thesystemsapproachisalsoevidentwhenitcomestodiscussingthestabilityoflearningalgorithmsandrecurrentnetworkretrievaldynamics,aswellasintheadoptedclassificationsofneuralnetworksasdiscrete-stateorcontinuous-stateanddiscrete-timeorcontinuous-time.

14、Theneuralnetworkparadigms(architecturesandtheirassociatedlearningrules)treatedherewereselectedbecauseoftheirrelevence,mathematicaltractability,and/orpracticality.Omissionshavebeenmadeforanumberofreasons,includingcomplexity,obscurity,andspace.Thisbookisorganizedintoeightchapters.Chapter1introducesthe

15、readertothemostbasicartificialneuralnet,consistingofasinglelinearthresholdgate(LTG).Thecomputationalcapabilitiesoflinearandpolynomialthresholdgatesarederived.Afundamentaltheorem,thefunctioncountingtheorem,isprovedandisappliedtostudythecapacityandthegeneralizationcapabilityofthresholdgates.Theconcept

16、scoveredinthischapterarecrucialbecausetheylaythetheoreticalfoundationsforjustifyingandexploringthemoregeneralartificialneuralnetworkarchitecturestreatedinlaterchapters.Chapter2mainlydealswiththeoreticalfoundationsofmultivariatefunctionapproximationusingneuralnetworks.Thefunctioncountingtheoremofchap

17、ter1isemployedtoderiveupperboundsonthecapacityofvariousfeedforwardnetsofLTGs.ThenecessaryboundsonthesizeofLTG-basedmultilayerclassifiersforthecasesoftrainingdataingeneralpositionandinarbitrarypositionarederived.Theoreticalresultsoncontinuousfunctionapproximationcapabilitiesoffeedforwardnets,withunit

18、semployingvariousnonlinearities,aresummarized.Thechapterconcludeswithadiscussionofthecomputationaleffectivenessofneuralnetarchitecturesandtheefficiencyoftheirhardwareimplementations.Learningrulesforsingle-unitandsingle-ayernetsarecoveredinChapter3.Morethan20basicdiscrete-timelearningrulesarepresente

19、d.Supervisedrulesareconsideredfirst,followedbyreinforcement,Hebbian,competitive,andfeaturemappingrules.Thepresentationoftheselearningrulesisunifiedinthesensethattheymayallbeviewedasrealizingincrementalsteepest-gradient-descentsearchonasuitablecriterionfunction.Examplesofsingle-layerarchitecturesareg

20、iventoillustratetheapplicationofunsupervisedlearningrules(e.g.,principalcomponentanalysis,clustering,vectorquantization,andself-organizingfeaturemaps).Chapter4isconcernedwiththetheoreticalaspectsofsupervised,unsupervised,andreinforcementlearningrules.Thechapterstartsbydevelopingaunifyingframeworkfor

21、thecharaterizationofvariouslearningrules(supervisedandunsupervised).Underthisframework,acontinuous-timelearningruleisviewedasafirst-orderstochasticdifferentialequation/dynamicalsystemwherebythestateofthesystemevolvessoas丫彳rptominimizeanassociatedinstantaneouscriterionfunction.Statisticalapproximatio

22、ntechniquesareemployedtostudythedynamicsandstability,inanaveragesense,ofthestochasticsystem.Thisapproximationleadstoanaveragelearningequationthat,inmostcases,canbecastasaglobally,asymptoticallystablegradientsystemwhosestableequilibriaareminimizersofawell-definedcriterionfunction.Formalanalysisisprov

23、idedforsupervised,reinforcement,Hebbian,competitive,andtopologypreservinglearning.Also,thegeneralizationpropertiesofdeterministicandstochasticneuralnetsareanalyzed.Thechapterconcludeswithaninvestigatinonofthecomplexityoflearninginmultilayerneuralnets.Chapter5dealswithlearninginmultilayerartificialne

24、uralnets.Itextendsthegradientdescent-basedlearningtomultilayerfeedforwardnets,whichresultsinthebackerrorpropagationlearningrule(orbackprop).Anextensivenumberofmethodsandheuristicsforimprovingbackpropsconvergencespeedandsolutionqualityarepresented,andanattemptismadetogiveatheoreticalbasisforsuchmetho

25、dsandheuristics.Severalsignificantapplicationsofbackprop-trainedmultilayernetsaredescribed.TheseapplicationsincludeconversionofEnglishtexttospeech,mappingofhandgesturestospeech,recognitionofhandwrittenZIPcodes,continuousvehiclenavigation,andmedicaldiagnosis.Thechapteralsoextendsbackproptorecurrentne

26、tworkscapableoftemporalassociation,nonlineardynamicalsystemmodeling,andcontrol.Chapter6isconcernedwithotherimportantadaptivemultilayernetarchitectures,suchastheradialbasisfunction(RBF)netandthecerebellermodelarticulationcontroller(CMAC)net,andtheirassociatedlearningrules.Thesenetworksoftenhavesimila

27、rcomputationalcapabilitiestofeedforwardmultilayernetsofsigmoidalunits,butwiththepotentialforfasterlearning.Adaptivemulilayerunit-allocatingnetssuchashypersphericalclassifiers,restrictedCoulombenergy(RCE)net,andcascadecorrelationnetarediscussed.Thechapteralsoaddressestheissueofunsupervisedlearninginm

28、ultilayernets,anditdescribestwospecificnetworksadaptiveresonancetheory(ART)netandtheautoassociativeclusteringnetsuitableforadaptivedataclustering.Theclusteringcapabilitiesofthesenetsaredemonstratedthroughexamples,includingthedecompositionofcomplexelectromyogramsignals.Chapter7discussesassociativeneu

29、ralmemories.Variousmodelsofassociativelearningandretrievalarepresentedandanalyzed,withemphasisonrecurrentmodels.Thestability,capacity,anderror-correctioncapabilitiesofthesemodelsareanalyzed.Thechapterconcludesbydescribingtheuseofoneparticularrecurrentmodel(theHopfieldcontinuousmodel)forsolvingcombin

30、atorialoptimizationproblems.GlobalsearchmethodsforoptimallearningandretrievalinmultilayerneuralnetworksisthetopicofChapter8.Itcoverstheuseofsimulatedannealing,meanfieldannealing,andgeneticalgorithmsforoptimallearning.Simulatedannealingisalsodiscussedinthecontextoflocal-minima-freeretrievalsinrecurre

31、ntneuralnetworks(Boltzmannmachines).Finally,ahybridgeneticalgorithm/gradient-descentsearchmethodthatcombinesoptimalandfastlearningisdescribed.Eachchapterconcludeswithasetofproblemsdesignedtoallowthereadertofurtherexploretheconceptsdiscussed.Morethan200problemsofvaryingdegreesofdifficultyareprovided.

32、Theproblemscanbedividedroughlyintothreecategories.Thefirstcategoryconsistsofproblemsthatarerelativelyeasytosolve.Theseproblemsaredesignedtodirectlyreinforcethetopicsdiscussedinthebook.Thesecondcategoryofproblems,markedwithanasterisk(*),isrelativelymoredifficult.Theseproblemsnormallyinvolvemathematic

33、alderivationsandproofsandareintendedtobethoughtprovoking.Manyoftheseproblemsincludereferencetotechnicalpapersintheliteraturethatmaygivecompleteorpartialsolutions.Thissecondcategoryofproblemsisintendedmainlyforreadersinterestedinexploringadvancedtopicsforthepurposeofstimulatingoriginalresearchideas.P

34、roblemsmarkedwithadagger()representathirdcategoryofproblemsthatarenumericalinnatureandrequiretheuseofacomputer.Someoftheseproblemsareminiprogrammingprojects,whichshouldbeespeciallyusefulforstudents.Thisbookcontainsenoughmaterialforafullsemestercourseonartificialneuralnetworksatthefirst-yeargraduatel

35、evel.Ihavealsousedthismaterialselectivelytoteachanupper-levelundergraduateintroductorycourse.Fortheundergraduatecourse,onemaychoosetoskipallorasubsetofthefollowingmaterial:Sections1.4-1.6,2.12.2,4.3-4.8,5.1.2,5.4.3-5.4.5,6.1.2,6.2-6.4,6.4.2,7.2.2,7.4.1-7.4.4,8.3.2,8.4.2,and8.6.hopethatthisbookwillpr

36、oveusefultothosestudentsandpracticingprofessionalswhoareinterestednotonlyinunderstandingtheunderlyingtheoryofartificialneuralnetworksbutalsoinpursuingresearchinthisarea.Alistofabout700releventreferencesisincludedwiththeaimofprovidingguidanceanddirectionforthereadersownsearchoftheresearchliterature.E

37、venthoughthisreferencelistmayseemcomprehensive,thepublishedliteratureistooextensivetoallowsuchalisttobecomplete.AcknowledgmentsFirstandforemost,Iacknowledgethecontributionsofthemanyresearchersintheareaofartificialneuralnetworksonwhichmostofthematerialinthistextisbased.Itwouldhavebeenextremelydifficu

38、lt(ifnotimpossible)towritethisbookwithoutthesupportandassistanceofanumberoforganizationsandindividuals.IwouldfirstliketothanktheNationalScienceFoundation(throughaPYIAward),ElectricPowerResearchInstitute(EPRI),FordMotorCompany,MentorGraphics,SunMicroSystems,UnisisCorporation,WhitakerFoundation,andZen

39、ithDataSystemsforsupportingmyresearch.IamalsogratefulforthesupportIhavereceivedforthisprojectfromWayneStateUniversitythroughaCareerDevelopmentChairAward.thankmystudents,whohavemadeclassroomuseofpreliminaryversionsofthisbookandwhosequestionsandcommentshavedefinitelyenhancedit.Inparticular,Iwouldliket

40、othankRaedAbuZitar,DavidClark,MikeFinta,JingSong,AgusSudjianto,Chuanming(Chuck)Wang,HuiWang,PaulWatta,andAbbasYoussef.IalsowouldliketothankmymanycolleaguesintheartificialneuralnetworkscommunityandatWayneStateUniversity,especiallyDr.A.RobertSpitzer,formanyenjoyableandproductiveconversationsandcollabo

41、rations.amindebttoMikeFinta,whoverycapablyandenthusiasticallytypedthecompletemanuscriptandhelpedwithmostoftheartwork,andtoDr.PaulWattaoftheComputationandNeuralNetworksLaboratory,WayneStateUniversity,forhiscriticalreadingofthemanuscriptandassistancewiththesimulationsthatledtoFigures5.3.8and5.3.9.Myde

42、epgratitudegoestothereviewersfortheircriticalandconstructivesuggestions.TheyareProfessorsShun-IchiAmarioftheUniversityofTokyo,JamesAndersonofBrownUniversity,ThomasCoverofStanfordUniversity,RichardGoldenoftheUniversityofTexas-Dallas,LaveenKanaloftheUniversityofMaryland,JohnTaylorofKingsCollegeLondon,

43、FrancisT.S.YuoftheUniversityofPennsylvania,Dr.GraninoKornofG.A.andT.M.KornIndustrialConsultants,andotheranonymousreviewers.Finally,letmethankmywifeAmal,daughterLamees,andsonTarekfortheirquietpatiencethroughthemanylonelyhoursduringthepreparationofthemanuscript.MohamadH.HassounDetroit,1994ProblemsProb

44、lemsTableofContentsFundamentalsofArtificialNeuralNetworksbyMohamadH.Hassoun(MITPress,1995)Chapter1ThresholdGates1.0IntroductionThresholdGatesLinearThresholdGatesQuadraticThresholdGatesPolynomialThresholdGatesComputationalCapabilitiesofPolynomialThresholdGatesGeneralPositionandtheFunctionCountingTheo

45、remWeierstrasssApproximationTheoremPointsinGeneralPositionFunctionCountingTheoremSeparabilityinf-SpaceMinimalPTGRealizationofArbitrarySwitchingFunctionsAmbiguityandGeneralizationExtremePointsSummaryProblemsChapter2ComputationalCapabilitiesofArtificialNeuralNetworks2.0IntroductionSomePreliminaryResul

46、tsonNeuralNetworkMappingCapabilitiesNetworkRealizationofBooleanFunctionsBoundsontheNumberofFunctionsRealizablebyaFeedforwardNetworkofLTGsNecessaryLowerBoundsontheSizeofLTGNetworksTwoLayerFeedforwardNetworksThreeLayerFeedforwardNetworksGenerallyInterconnectedNetworkswithnoFeedbackApproximationCapabil

47、itiesofFeedforwardNeuralNetworksforContinuousFunctionsKolmogorovsTheoremSingleHiddenLayerNeuralNetworksareUniversalApproximatorsSingleHiddenLayerNeuralNetworksareUniversalClassifiersComputationalEffectivenessofNeuralNetworksAlgorithmicComplexityComputationalEnergySummaryChapter3LearningRules3.0Intro

48、ductionSupervisedLearninginaSingleUnitSettingErrorCorrectionRulesPerceptronLearningRuleGeneralizationsofthePerceptronLearningRuleThePerceptronCriterionFunctionMaysLearningRuleWidrow-Hoff(alpha-LMS)LearningRuleOtherGradientDescent-BasedLearningRulesmu-LMSLearningRuleThemu-LMSasaStochasticProcessCorre

49、lationLearningRuleExtensionofthemu-LMSRuletoUnitswithDifferentiableActivationFunctions:DeltaRuleAdaptiveHo-Kashyap(AHK)LearningRulesOtherCriterionFunctionsExtensionofGradientDescent-BasedLearningtoStochasticUnitsReinforcementLearningAssociativeReward-PenaltyReinforcementLearningRuleUnsupervisedLearn

50、ingHebbianLearningOjasRuleYuilleetal.RuleLinskersRuleHebbianLearninginaNetworkSetting:PrincipalComponentAnalysis(PCA)PCAinaNetworkofInteractingUnitsPCAinaSingleLayerNetworkwithAdaptiveLateralConnectionsNonlinearPCACompetitivelearningSimpleCompetitiveLearningVectorQuantizationSelf-OrganizingFeatureMa

51、ps:TopologyPreservingCompetitiveLearningKohonensSOFMExamplesofSOFMsSummaryProblemsChapter4MathematicalTheoryofNeuralLearning4.0IntroductionLearningasaSearchMechanismMathematicalTheoryofLearninginaSingleUnitSettingGeneralLearningEquationAnalysisoftheLearningEquationAnalysisofsomeBasicLearningRulesCha

52、racterizationofAdditionalLearningRulesSimpleHebbianLearningProblemsProblemsImprovedHebbianLearningOjasRuleYuilleetal.RuleHassounsRulePrincipalComponentAnalysis(PCA)TheoryofReinforcementLearningTheoryofSimpleCompetitiveLearningDeterministicAnalysisStochasticAnalysisTheoryofFeatureMappingCharacterizat

53、ionofKohonensFeatureMapSelf-OrganizingNeuralFieldsGeneralizationGeneralizationCapabilitiesofDeterministicNetworksGeneralizationinStochasticNetworksComplexityofLearningSummaryProblemsChapter5AdaptiveMultilayerNeuralNetworksI5.0IntroductionLearningRuleforMultilayerFeedforwardNeuralNetworksErrorBackpro

54、pagationLearningRuleGlobalDescent-BasedErrorBackpropagationBackpropEnhancementsandVariationsWeightsInitializationLearningRateMomentumActivationFunctionWeightDecay,WeightElimination,andUnitEliminationCross-ValidationCriterionFunctionsApplicationsNetTalkGlove-TalkHandwrittenZIPCodeRecognitionALVINN:AT

55、rainableAutonomousLandVehicleMedicalDiagnosisExpertNetImageCompressionandDimensionalityReductionExtensionsofBackpropforTemporalLearningTime-DelayNeuralNetworksBackpropagationThroughTimeRecurrentBack-PropagationTime-DependentRecurentBack-Propagation545Real-TimeRecurrentLearningSummaryChapter6Adaptive

56、MultilayerNeuralNetworksII6.0IntroductionRadialBasisFunction(RBF)NetworksRBFNetworksversusBackpropNetworksRBFNetworkVariationsCerebellerModelArticulationController(CMAC)CMACRelationtoRosenblattsPerceptronandOtherModelsUnit-AllocatingAdaptiveNetworksHypersphericalClassifiersRestrictedCoulombEnergy(RC

57、E)ClassifierReal-TimeTrainedHypersphericalClassifierCascade-CorrelationNetworkClusteringNetworks641AdaptiveResonanceTheory(ART)NetworksAutoassociativeClusteringNetworkSummaryProblemsChapter7AssociativeNeuralMemories7.0IntroductionBasicAssociativeNeuralMemoryModelsSimpleAssociativeMemoriesandtheirAss

58、ociatedRecordingRecipesCorrelationRecordingRecipeASimpleNonlinearAssociativeMemoryModelOptimalLinearAssociativeMemory(OLAM)OLAMErrorCorrectionCapabilitiesStrategiesforImprovingMemoryRecordingDynamicAssociativeMemories(DAM)Continuous-TimeContinuous-StateModelDiscrete-TimeContinuous-StateModelDiscrete

59、-TimeDiscrete-StateModelDAMCapacityandRetrievalDyanamicsCorrelationDAMsProjectionDAMsCharacteristicsofHigh-PerformanceDAMsOtherDAMModelsBrain-State-in-a-Box(BSB)DAMNon-MonotonicActivationsDAMDiscreteModelContinuousModelHystereticActivationsDAMExponentialCapacityDAMSequenceGeneratorDAMHeteroassociati

60、veDAMTheDAMasaGradientNetanditsApplicationtoCombinatorialOptimizationSummaryChapter8GlobalSearchMethodsforNeuralNetworks1.8.0IntroductionLocalversusGlobalSearchAGradientDescent/AscentSearchStrategyStochasticGradientSearch:GlobalSearchviaDiffusionSimulatedAnnealing-BasedGlobalSearchSimulatedAnnealing