人工神经网络

1、人工神经网络,中国科学院自动化研究所 吴高巍 2016-11-29,联结主义学派,又称仿生学派或生理学派 认为人的思维基元是神经元，而不是符号处理过程 认为人脑不同于电脑 核心：智能的本质是联接机制。 原理：神经网络及神经网络间的连接机制和学习算法,麦卡洛可（McCulloch）,皮茨（Pitts),什么是神经网络,所谓的人工神经网络就是基于模仿生物大脑的结构和功能而构成的一种信息处理系统（计算机）。 个体单元相互连接形成多种类型结构的图 循环、非循环 有向、无向 自底向上（Bottom-Up）AI 起源于生物神经系统 从结构模拟到功能模拟,仿生,人工神经网络,内容,生物学启示 多层神经网络

2、Hopfield网络 自组织网络,生物学启示,神经元组成：细胞体，轴突，树突，突触 神经元之间通过突触两两相连。信息的传递发生在突触。 突触记录了神经元间联系的强弱。 只有达到一定的兴奋程度，神经元才向外界传输信息。,生物神经元,神经元,神经元特性 信息以预知的确定方向传递 一个神经元的树突细胞体轴突突触另一个神经元树突 时空整合性 对不同时间通过同一突触传入的信息具有时间整合功能 对同一时间通过不同突触传入的信息具有空间整合功能,神经元,工作状态 兴奋状态，对输入信息整合后使细胞膜电位升高，当高于动作电位的阈值时，产生神经冲动，并由轴突输出。 抑制状态，对输入信息整合后使细胞膜电位降低，当低

3、于动作电位的阈值时，无神经冲动产生。 结构的可塑性 神经元之间的柔性连接：突触的信息传递特性是可变的 学习记忆的基础,神经元模型,从生物学结构到数学模型,人工神经元,M-P模型,McClloch and Pitts, A logical calculus of the ideas immanent in nervous activity, 1943,f: 激活函数(Activation Function) g: 组合函数(Combination Function),Weighted Sum Radial Distance,组合函数,(e) (f),Threshold,Linear,Satura

4、ting Linear,Logistic Sigmoid,Hyperbolic tangent Sigmoid,Gaussian,激活函数,人工神经网络,多个人工神经元按照特定的网络结构联接在一起，就构成了一个人工神经网络。 神经网络的目标就是将输入转换成有意义的输出。,生物系统中的学习,自适应学习 适应的目标是基于对环境信息的响应获得更好的状态 在神经层面上，通过突触强度的改变实现学习 消除某些突触，建立一些新的突触,生物系统中的学习,Hebb学习律 神经元同时激活，突触强度增加 异步激活，突触强度减弱 学习律符合能量最小原则 保持突触强度需要能量，所以在需要的地方保持，在不需要的地方不保持

5、。,ANN的学习规则,能量最小 ENERGY MINIMIZATION 对人工神经网络，需要确定合适的能量定义；可以使用数学上的优化技术来发现如何改变神经元间的联接权重。,ENERGY = measure of task performance error,两个主要问题 结构 How to interconnect individual units? 学习方法 How to automatically determine the connection weights or even structure of ANN?,Solutions to these two problems leads

6、to a concrete ANN!,人工神经网络,前馈结构(Feedforward Architecture) - without loops - static 反馈/循环结构(Feedback/Recurrent Architecture) - with loops - dynamic (non-linear dynamical systems),ANN结构,General structures of feedforward networks,General structures of feedback networks,通过神经网络所在环境的模拟过程，调整网络中的自由参数 Learnin

7、g by data 学习模型 Incremental vs. Batch 两种类型 Supervised vs. Unsupervised,ANN的学习方法,若两端的神经元同时激活，增强联接权重 Unsupervised Learning,学习策略: Hebbrian Learning,最小化实际输出与期望输出之间的误差(Supervised) - Delta Rule (LMS Rule, Widrow-Hoff) - B-P Learning,Objective:,Solution:,学习策略: Error Correction,采用随机模式，跳出局部极小 - 如果网络性能提高，新参数被接

8、受. - 否则，新参数依概率接受,学习策略: Stochastic Learning,“胜者为王”(Winner-take-all ) Unsupervised How to compete? - Hard competition Only one neuron is activated - Soft competition Neurons neighboring the true winner are activated.,学习策略: Competitive Learning,重要的人工神经网络模型,多层神经网络 径向基网络 Hopfield网络 Boltzmann机 自组织网络 ,多层感知

9、机(MLP),感知机实质上是一种神经元模型 阈值激活函数,Rosenblatt, 1957,感知机,判别规则,输入空间中 样本是空间中的一个点 权向量是一个超平面 超平面一边对应 Y=1 另一边对应 Y=-1,单层感知机学习,调整权值，减少训练集上的误差 简单的权值更新规则: 初始化 对每一个训练样本: Classify with current weights If correct, no change! If wrong: adjust the weight vector,30,学习: Binary Perceptron,初始化 对每一个训练样本: Classify with curren

10、t weights If correct (i.e., y=y*), no change! If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1.,多类判别情况,If we have multiple classes: A weight vector for each class: Score (activation) of a class y: Prediction highest score wins,学习: Multiclass Percep

11、tron,初始化 依次处理每个样本 Predict with current weights If correct, no change! If wrong: lower score of wrong answer, raise score of right answer,感知机特性,可分性: true if some parameters get the training set perfectly correct Can represent AND, OR, NOT, etc., but not XOR 收敛性: if the training is separable, perceptr

12、on will eventually converge (binary case),Separable,Non-Separable,感知机存在的问题,噪声（不可分情况）: if the data isnt separable, weights might thrash 泛化性: finds a “barely” separating solution,改进感知机,线性可分情况,Which of these linear separators is optimal?,Support Vector Machines,Maximizing the margin: good according to

13、intuition, theory, practice Only support vectors matter; other training examples are ignorable Support vector machines (SVMs) find the separator with max margin,SVM,优化学习,问题描述 训练数据 目标：发现最好的权值，使得对每一个样本x的输出都符合类别标签 样本xi的标签可等价于标签向量 采用不同的激活函数 平方损失：,单层感知机,单层感知机,单层感知机,单层感知机,采用线性激活函数，权值向量具有解析解 批处理模式 一次性更新权重

14、缺点：收敛慢 增量模式 逐样本更新权值 随机近似，但速度快并能保证收敛,多层感知机 (MLP),层间神经元全连接,MLPs表达能力,3 layers: All continuous functions 4 layers: all functions,How to learn the weights?,waiting B-P algorithm until 1986,B-P Network,结构 A kind of multi-layer perceptron, in which the Sigmoid activation function is used.,B-P 算法,学习方法 - Inp

15、ut data was put forward from input layer to hidden layer, then to out layer - Error information was propagated backward from out layer to hidder layer, then to input layer,Rumelhart & Meclelland, Nature,1986,B-P 算法,Global Error Measure,desired output,generated output,squared error,The objective is t

16、o minimize the squared error, i.e. reach the Minimum Squared Error (MSE),B-P 算法,Step1. Select a pattern from the training set and present it to the network. Step2. Compute activation of input, hidden and output neurons in that sequence. Step3. Compute the error over the output neurons by comparing t

17、he generated outputs with the desired outputs. Step4. Use the calculated error to update all weights in the network, such that a global error measure gets reduced. Step5. Repeat Step1 through Step4 until the global error falls below a predefined threshold.,梯度下降方法,Optimization method for finding out

18、the weight vector leading to the MSE,learning rate,gradient,vector form:,element form:,权值更新规则,For output layer:,权值更新规则,For output layer:,权值更新规则,For hidden layer,权值更新规则,For hidden layer,应用: Handwritten digit recognition,3-nearest-neighbor = 2.4% error 40030010 unit MLP = 1.6% error LeNet: 768 192 30

19、10 unit MLP = 0.9% error Current best (SVMs) 0.4% error,MLPs：讨论,实际应用中 Preprocessing is important Normalize each dimension of data to -1, 1 Adapting the learning rate t = 1/t,MLPs：讨论,优点： 很强的表达能力 容易执行 缺点： 收敛速度慢 过拟合（Over-fitting） 局部极小,采用Newton法,加正则化项，约束权值的平滑性 采用更少（但足够数量）的隐层神经元,尝试不同的初始化 增加扰动,Hopfield 网络

20、,反馈 结构 可用加权无向图表示 Dynamic System 两种类型 Discrete (1982) and Continuous (science, 1984), by Hopfield,Hopfield网络,Combination function：Weighted Sum Activation function：Threshold,吸引子与稳定性,How do we “program” the solutions of the problem into stable states (attractors) of the network? How do we ensure that t

21、he feedback system designed is stable? Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector, called a Lyapunov (Energy) Function.,Hopfield网络的能量函数,With input Without input,Hopfield 模型,Hopfield证明了异步Hopfield网络

22、是稳定的，其中权值定义为 Whatever be the initial state of the network, the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.,Hopfield 网络: 联想记忆,Hopfield网络的一个主要应用 基于与数据部分相似的输入，可以回想起数据本身（attractor state） 也称作内容寻址记忆(content-addressable memory).,Stored

23、 Pattern,Memory Association,虞台文, Feedback Networksand Associative Memories,Hopfield 网络: Associative Memories,Stored Pattern,Memory Association,虞台文, Feedback Networksand Associative Memories,Hopfield网络的一个主要应用 基于与数据部分相似的输入，可以回想起数据本身（attractor state） 也称作内容寻址记忆(content-addressable memory).,How to store

24、patterns?,=?,How to store patterns?,=?,: Dimension of the stored pattern,权值确定: 外积(Outer Product),Vector form: Element form: Why? Satisfy the Hopfield model,An example of Hopfield memory,虞台文, Feedback Networks and Associative Memories,Stable,E=4,E=0,E=4,Recall the first pattern (x1),Stable,E=4,E=0,E=

25、4,Recall the second pattern (x2),Hopfield 网络: 组合优化(Combinatorial Optimization),Hopfield网络的另一个主要应用 将优化目标函数转换成能量函数(energy function) 网络的稳定状态是优化问题的解,例: Solve Traveling Salesman Problem (TSP),Given n cities with distances dij, what is the shortest tour?,Illustration of TSP Graph,1,2,3,4,5,6,7,8,9,10,11,H

26、opfield Network for TSP,=?,Hopfield Network for TSP,=,City matrix,Constraint 1. Each row can have only one neuron “on”. 2. Each column can have only one neuron “on”. 3. For a n-city problem, n neurons will be on.,Hopfield Network for TSP,1,2,4,3,5,Time,City,The salesman reaches city 5 at time 3.,Wei

27、ght determination for TSP: Design Energy Function,Constraint-1,Constraint-2,Constraint-3,能量函数转换为2DHopfield网络形式,Network is built!,Hopfield网络迭代（TSP ）,The initial state generated randomly goes to the stable state (solution) with minimum energy,A 4-city example 阮晓刚， 神经计算科学，2006,自组织特征映射 (SOFM),What is SO

28、FM?,Neural Network with Unsupervised Learning Dimensionality reduction concomitant with preservation of topological information. Three principals - Self-reinforcing - Competition - Cooperation,Structure of SOFM,竞争(Competition),Finding the best matching weight vector for the present input. Criterion for determining the winning neuron: Maximum Inner Product Minimum Euclidean Distance,合作(Coop