生物神经网络

1、2022/7/231Chapter 3 NEURONAL DYNAMICS II:ACTIVATION MODELSPart I2022/7/232生物神经网人工神经元人工神经网络的拓扑特性 神经动力学系统概念 人工神经网络通用的信号函数神经元的存贮模式上次课程回顾2022/7/233胞体(Soma)树突（Dendrite）胞体(Soma) 轴突（Axon）突触（Synapse）生物神经网络2022/7/234生物神经网的六个基本特征：1）神经元及其联接；2）神经元之间的联接强度决定信号传递的强弱；3）神经元之间的联接强度是可以随训练改变的；4）信号可以是起刺激作用的，也可以是起抑制作用的；5

2、）一个神经元接受的信号的累积效果决定该神经元的状态；6) 每个神经元可以有一个“阈值”。2022/7/235Cell body (soma)DendriteNucleus神经元 (Neuron)AxonSynapse树突(Dendtrite)：输入端 轴突(Axon)： 输出端突触(Synapse)： 不同神经元的轴突与树突的结合部，不同神 经元的相互作用用权值表示，学习就是调整权值，胞体(Soma)： 是非线性输入/输出的单元，可用阈值、分段、Sigmod函 数近似从仿生学角度：2022/7/236人工神经网络2022/7/237人工神经元 神经元是构成神经网络的最基本单元（构件）。人工神经

3、元模型应该具有生物神经元的六个基本特性。 2022/7/238人工神经元的基本构成 人工神经元模拟生物神经元的一阶特性。输入：X=（x1，x2，xn）联接权：W=（w1，w2，wn）T网络输入：net=xiwi向量形式：net=XWxn wnx1 w1x2 w2net=XW2022/7/239激活函数(Activation Function) 激活函数执行对该神经元所获得的网络输入的变换，也可以称为激励函数、活化函数： o=f（net） 1、线性函数（Liner Function） f（net）=k*net+c netooc2022/7/23102、非线性斜面函数(Ramp Function)

4、 if netf（net）= k*netif |net|0为一常数，被称为饱和值，为该神经元的最大输出。 2022/7/2311 - - net o 2022/7/23123、阈值函数（Threshold Function） 或 阶跃函数if netf（net）=-if net 、均为非负实数，为阈值二值形式：1if netf（net）=0if net 双极形式：1if netf（net）=-1if net 2022/7/2313 -onet02022/7/23144、S形函数 压缩函数（Squashing Function）和逻辑斯特函数（Logistic Function）。f（net）=

5、a+b/(1+exp(-d*net)a，b，d为常数。它的饱和值为a和a+b。最简单形式为：f（net）= 1/(1+exp(-d*net) 函数的饱和值为0和1。S形函数有较好的增益控制 2022/7/2315a+b o(0,c)netac=a+b/22022/7/2316二值函数双极函数S型函数线性函数2022/7/2317空间模式（Spatial Model）时空模式（Spatialtemporal Model）空间模式三种存储类型1、 RAM方式（Random Access Memory）随机访问方式是将地址映射到数据。2、 CAM方式（Content Addressable Memo

6、ry）内容寻址方式是将数据映射到地址。3、 AM方式（Associative Memory）相联存储(联想记忆)方式是将数据映射到数据。 神经元的存贮模型2022/7/2318后两种方式是人工神经网络的工作方式。在学习/训练期间，人工神经网络以CAM方式工作；权矩阵又被称为网络的长期存储（Long Term Memory，简记为LTM）。网络在正常工作阶段是以AM方式工作的；神经元的状态表示的模式为短期存储（Short Term Memory，简记为STM）。 2022/7/2319自相联（Auto-associative）映射：训练网络的样本集为向量集合为 A1，A2，An在理想情况下，该网

7、络在完成训练后，其权矩阵存放的将是上面所给的向量集合。 2022/7/2320异相联（Hetero-associative）映射（A1，B1），（A2，B2），（An，Bn） 该网络在完成训练后，其权矩阵存放的将是上面所给的向量集合所蕴含的对应关系。当输入向量A不是样本的第一的分量时，样本中不存在这样的元素（Ak，Bk），使得AiAkA或者AAkAj且此时有AiAAj则向量B是Bi与Bj的插值。 2022/7/2321本节内容安排神经动力学系统BAM模型Lypaunov函数Bivalent(二阶) BAM理论2022/7/2322神经元激励函数随时间变化，其改变方式依赖于下面的动力学方程：(3

8、-1)(3-2)3.1 神经动力学系统2022/7/2323Na+K+Cl-Cl-K+Organic ionNa+30 mV-+Na+K+Cl-Na+Cl-K+Organic ion+-70 mV-+细胞膜内外电动势神经动力系统在神经元中的体现2022/7/2324在缺乏外部作用和神经刺激的情况下，最简单激励的动力学模型是：(3-3)(3-4) 一阶被动衰减模型2022/7/2325在任何有穷状态下膜电压以指数方式迅速衰减到零电位。2022/7/2326 Passive Membrane Decay（被动膜电压的衰减） Passive-decay ratescales the rate to

9、the membranes resting potential. solution :Passive-decay rate measures: the cell membranesresistance or “friction” to current flow.2022/7/2327The larger the passive-decay rate,the faster the decay-the less the resistance to current flow. Pay attention to propertyproperty2022/7/2328 Membrane Time Con

10、stants The membrane time constant scales the time variable of the activation dynamical system.The multiplicative constant model:（38） 2022/7/2329Solution and propertysolutionpropertyThe smaller the capacitance ,the faster things change As the membrane capacitance increases toward positive infinity,me

11、mbrane fluctuation slows to stop.2022/7/2330Membrane Resting PotentialsDefine resting Potential as the activation value to which the membrane potential equilibrates in the absence of external or neuronal inputs: Solutions Definition(3-11)(3-12)2022/7/2331 Note The capacitance appear in the index of

12、the solution,it is called time-scaling capacitance. It does no affect the steady-state solution and does not depend on the finite initial condition. In resting case,we can find the solution quickly.2022/7/2332Additive External InputAdd inputApply a relatively constant numeral input to a neuron. solu

13、tion(3-13)(3-14)2022/7/2333Meaning of the input Input can represent the magnitude of directly experiment sensory information or directly apply control information. The input changes slowly,and can be assumed constant value. 2022/7/2334神经网络模型 神经网络模型已有40多种，代表性的有：自适应共振(ART)雪崩网络双向联想记忆(BAM)BP模型Boltzman机/

14、Cauchy机(BCM)盒中脑(BSB)Counter Propagation(CPN)Hopfield神经网络Madaline学习矩阵(LRN)自组织映射(SOM)细胞神经网络(CNN)交替投影神经网络(APNN)有关42种模型的详细说明请参见神经网络计算焦李成编（1996）P36P902022/7/2335Bart.Kosko对ANS领域作出了三大贡献：双向联想记忆模型（BAM）模糊认知映射模糊逻辑与ANS结合BAM模型将单层的Hebb学习器推广成两层的模型匹配互联想器，BAM是非监督学习的、能够实时学习和回忆的ANS。由于其反馈结构及描述其两层之间相互作用的动力学方程的性质，BAM能够在

15、任何给定的矩阵后收敛于一个最小解，使BAM能够建立一种能够同时学习和回忆的全局稳定的动力学系统。Bart.Kosko对ANS的贡献2022/7/2336What is Associative Memory and BAM ? Associative memory（联想记忆）：是神经网络的重要应用之一，它具有很强的容错性和抗干扰性，能进行大规模并行处理，适用于模式识别、信号处理、故障诊断和图像处理等。 Bidirectional Associative Memory (BAM)：中国首次载人航天时间：2003.10.15地点：甘肃九泉发射场航天员：杨利伟 2022/7/2337BAM结构 智力链

16、从一件事想到另一件事，“唤回失去的记忆”。自相联异相联双向联想记忆BAM（Bidirectional Associative Memory）。BAM具有一定的推广能力它对含有一定缺陷的输入向量，通过对信号的不断变换、修补，最后给出一个正确的输出。 2022/7/2338基本的BAM结构 W第1层输入向量第2层输出向量WTx1xnymy12022/7/2339The BAM : Network ArchitecturexFyxF-1yOutput layerInput layerFeedback partFeedforwardpart2022/7/2340Kosko的改进型BAM 离散BAM连续

17、BAM自适应BAM高阶自适应BAM竞争自适应BAM随机自适应BAM模糊自适应BAM2022/7/2341 Neurons do not compute alone. Neuron modify their state activations with external input and with the feedback from one another. 3.2 ADDITIVE NEURONAL FEEDBACK This feedback takes the form of path-weighted signals from synaptically connected neuron

18、s.2022/7/2342Synaptic Connection Matricesn neurons in field p neurons in field The ith neuron axon in a synapse jth neurons in is constant,can be positive,negative or zero.2022/7/2343Meaning of connection matrix The synaptic matrix or connection matrix M is an n-by-p matrix of real number whose entr

19、ies are the synaptic efficacies .the ijth synapse is excitatoryif ，inhibitory（抑制） if The matrix M describes the forward projections from neuron field to neuron field The matrix N describes the feedforward projections from neuron field to neuron field2022/7/2344Bidirectional and Unidirectional connec

20、tion TopologiesBidirectional networksM and N have the same or approximately the samestructure. Unidirectional networkA neuron field synaptically intraconnects to itself. BAM M is symmetric, the unidirectional network is BAM2022/7/2345Augmented field and augmented matrix Augmented fieldM connects to

21、,N connects to then the augmented field intraconnects to itself by the square block matrix B2022/7/2346Augmented field and augmented matrix In the BAM case,when then hence a BAM symmetries an arbitrary rectangular（长方形） matrix M. In the general case,P is n-by-n matrix.Q is p-by-p matrix.If and only i

22、f, the neurons in are symmetrically intraconnected TCC=2022/7/23473.3 ADDITIVE ACTIVATION MODELSDefine additive activation modeln+p coupled first-order differential equations defines the additive activation model（315）（316） 2022/7/2348additive activation model define The additive autoassociative mode

23、l correspond to a system of n coupled first-order differential equations （317）2022/7/2349additive activation model define A special case of the additive autoassociative model(3-18)(3-19)（3-20）where measures the cytoplasmic resistance between neurons i and j. is2022/7/2350Hopfield circuit and continu

24、ous additive bidirectionalassociative memories Hopfield circuit arises from if each neuron has a strictly increasing signal function and if the synaptic connection matrix is symmetric(3-21) continuous additive bidirectional associative memories(3-22)(3-23)2022/7/2351 Discrete additive activation mod

25、els correspond to neurons with threshold signal function3.4 ADDITIVE BIVALENT FEEDBACK The neurons can assume only two value: ON and OFF. ON represents the signal value +1. OFF represents 0 or 1. Bivalent models can represent asynchronous and stochastic （随机的）behavior.2022/7/2352Bivalent Additive BAM

26、 BAM-bidirectional associative memory Define a discrete additive BAM with threshold signal functions, arbitrary thresholds and inputs,an arbitrary but constant synaptic connection matrix M,and discrete time steps k. (3-24)(3-25)2022/7/2353Bivalent Additive BAMThreshold binary signal functions For ar

27、bitrary real-value thresholds for neurons for neurons(3-26)(3-27)2022/7/2354A example for BAM modelExampleA 4-by-3 matrix M represents the forward synaptic projections from to .A 3-by-4 matrix MT represents the backward synaptic projections from to .2022/7/2355A example for BAM model (cont.)Suppose

28、at initial time k all the neurons in are ON.So the signal state vector at time k corresponds toInputSuppose2022/7/2356A example for BAM model (cont.) first:at time k+1 through synchronous operation,the result is: next:at time k+1 ,these signals pass “forward” through the filter M to affect the activ

29、ations of the neurons.The three neurons compute three dot products,or correlations.The signal state vector multiplies each of the three columns of M.2022/7/2357A example for BAM model (cont.) the result is: synchronously compute the new signal state vector :2022/7/2358A example for BAM model (cont.)

30、 the signal vector passes “backward” through the synapticfilter at time k+2: synchronously compute the new signal state vector :2022/7/2359A example for BAM model (cont.) conclusionsincethenThese same two signal state vectors will pass back and forth in bidirectional equilibrium forever-or until new

31、 inputs perturb the system out of equilibrium.2022/7/2360A example for BAM model (cont.) asynchronous state changes may lead to different bidirectional equilibrium keep the first neurons ON,only update the second and third neurons. At k,all neurons are ON. new signal state vector at time k+1 equals:

32、2022/7/2361A example for BAM model (cont.) new activation state vector equals: synchronously thresholds passing this vector forward to gives2022/7/2362A example for BAM model (cont.) similarly, for any asynchronous state change policy we apply to the neurons the system has reached a new equilibrium,

33、the binary pair represents a fixed point of the system. 2022/7/2363conclusion conclusion Different subset asynchronous state change policies applied to the same data need not product the same fixed-point equilibrium. They tend to produce the same equilibria.All BAM state changes lead to fixed-point

34、stability.2022/7/2364The BAM Examplein MATLABx1 = 1 -1 -1 1 -1 1 1 -1 -1 1;y1 = 1 -1 -1 -1 -1 1;x2 = 1 1 1 -1 -1 -1 1 1 -1 -1;y2 = 1 1 1 1 -1 -1;w = y1*x1 + y2*x2Matlab codeExamplar No. 1Examplar No. 2Weight matrix 2 0 0 0 -2 0 2 0 -2 0 0 2 2 -2 0 -2 0 2 0 -2 0 2 2 -2 0 -2 0 2 0 -2 0 2 2 -2 0 -2 0 2

35、 0 -2 -2 0 0 0 2 0 -2 0 2 0 0 -2 -2 2 0 2 0 -2 0 2w =2022/7/2365The BAM : Example (cont.)x0 = -1 -1 -1 1 -1 1 1 -1 -1 1;y0 = 1 1 1 1 -1 -1;xold = x0; yold = y0;for i=1:5 nety = w*xold ynew = (nety0)*1 + (nety=0).*yold - (nety0)*1 + (netx=0).*xold - (netx0)*1 if (xnew=xold) break; end xold = xnew; yo

36、ld = ynew; pauseendUnknown inputSet initial valuesFeedforward partFeedback partUpdate value forNext roundStop if no change2022/7/2366The BAM : Example (cont.)Iteration 1: 4 -12 -12 -12 -4 12nety = 1 -1 -1 -1 -1 1ynew = 4 -8 -8 8 -4 8 4 -8 -4 8netx = 1 -1 -1 1 -1 1 1 -1 -1 1xnew = Final state : xnew

37、= x12022/7/2367Bidirectional Stability definition A BAM system is Bidirectional stable if all inputs converge to fixed-point equilibria.A denotes a binary n-vector in B denotes a binary p-vector in 2022/7/2368Bidirectional Stability (cont.)Represent a BAM system equilibrates to bidirectional fixed p

38、oint as2022/7/2369Lyapunov第二方法是建立在客观事实上：如果一个系统的某个 平衡状态是是渐近稳定的，即 则随着系统的运动，其贮存的能量将随着时间的增长而衰减，直至趋于平衡状态而使能量趋于极小值。但对一般而言，并没有这样的直观性，因此Lyapunov引入了一个广义的能量函数，称为Lyapunov函数。 ，其中p为对称矩阵，这样Lyapunov第二方法就归结为：在不直接求解X的前提下通过研究V函数及其导数就可以给出系统平衡状态的稳定信息。3.5 Lyapunov Functions(李雅普诺夫函数)2022/7/2370Lyapunov Functions L maps s

39、ystem state variables to real numbers and decreases with time. In BAM case,L maps the Bivalent product space to real numbers.Suppose L is sufficiently differentiable（充分可微） to apply the chain rule:(3-28)2022/7/2371 Lyapunov Functions (cont.) The quadratic choice of L Suppose the dynamical system desc

40、ribes the passive decay system.（3-29）（3-30） The solution（3-31）2022/7/2372Lyapunov Functions (cont.) The partial derivative（偏导数） of the quadratic L:（3-32）（3-33）(3-34)or(3-35)In either caseAt equilibrium(3-36)This occurs if and only if all velocities equal zero2022/7/2373conclusion A dynamical system

41、is stable if some Lyapunov Functions Ldecreases along trajectories. A dynamical system is asymptotically stable if it strictly decreases along trajectories Monotonicity of a Lyapunov Function provides a sufficient condition for stability and asymptotic stability.2022/7/2374Linear system stabilityFor

42、 symmetric matrix A and square matrix B,the quadratic form behaves as a strictly decreasing Lyapunovfunction for any linear dynamical system if and only if the matrix is negative definite.2022/7/2375The relations between convergence rate and eigenvalue sign （特征值符号） A general theorem in dynamical sys

43、tem theory relates convergence rate and ergenvalue sign: A nonlinear dynamical system converges exponetially（指数） quickly if its system Jacobian has eigenvalues with negative real parts. Locally such nonlinear system behave as linearly. A Lyapunov Function summarizes total system behavuor. A Lyapunov

44、 Function often measures the energy of a physical sysem. 2022/7/2376Potential energy function represented by quadratic formConsider a system of n variables and its potential-energy function E. Suppose the coordinate measures the displacement from equilibrium of ith unit.The energy depends on only co

45、ordinate ,so since E is a physical quantity,we assume it is sufficiently smooth to permit a multivariable Taylor-series expansion about the origin: 2022/7/2377Potential energy function represented by quadratic form (cont.)Where A is symmetric,since2022/7/2378The reason of (3-42)followsFirst,we defin

46、ed the origin as an equilibrium of zero potentialenergy;so Second,the origin is an equilibrium only if all first partialderivatives equal zero.Third,we can neglect higher-order terms for small displacement,since we assume the higher-order products are smaller than the quadratic products.2022/7/23793

47、.6 Bivalent BAM theoremThe average signal energy L of the forward pass of the Signal state vector through M,and the backward passOf the signal state vector through : since2022/7/2380Lower bound of Lyapunov function The signal is Lyapunov function clearly bounded below.For binary or bipolar,the matri

48、x coefficients（系数） definethe attainable bound:The attainable upper bound is the negative of this expression.2022/7/2381Lyapunov function for the general BAM system The signal-energy Lyapunov function for the general BAM system takes the formInputs and andconstant vectors of thresholdsthe attainable bound of this function is.2022/7/2382Bivalent BAM theoremBivalent BAM theorem.every matrix is bidrectionally stablefor synchronous or asynchronous state chang