《MLP神经网络》PPT课件.ppt

Multi-layer Perceptrons Junying Zhang contents structure universal theorem MLP for classification mechanism of MLP for classification nonlinear mapping binary coding of the areas MLP for regression learning algorithm of the MLP back propagation learning algorithm heuristics in learning process XOR and Linear Separability Revisited Remember that it is not possible to find weights that enable Single Layer Perceptrons to deal with non-linearly separable problems like XOR: However, Multi-Layer Perceptrons (MLPs) are able to cope with non-linearly separable problems. Historically, the problem was that there were no learning algorithms for training MLPs. Actually, it is now quite straightforward. Structure of an MLP it is composed of several layers neurons within each layer are not connected ith layer is only fully connected to the (i+1)jth layer Signal is transmitted only in a feedforward manner Structure of an MLP Model of each neuron in the net includes A nonlinear activation function the net is nonlinear The function is smooth derivative Generally, sigmoidal function The network contains one or more layers of hidden neurons that are not part of input or output of the net enable the net to learn complex tasks Expressive power of an MLP Questions How many hidden layers are needed? How many units should be in a (the) hidden layer? Answers Komogorovs mapping neural network existence theorem (universal theorem) Komogorovs mapping neural network existence theorem (universal theorem) Any continuous function g(x) defined on the unit hypercube can be represented in the from For properly chosen functions and It is impractical the functions and are not the simple weighted sums passed through nonlinearities favored in neural networks It tells us very little about how to find the nonlinear functions based on data the central problem in network based pattern recognition those functions can be extremely complex; they are not smooth Komogorovs mapping neural network existence theorem (universal theorem) Any continuous function g(x) can be approximated to arbitrary precision by for properly chosen function f(.) when NH approaches to infinity. MLP for classification MLP for regression Learning scheme Supervised learning Two propagation directions - Function Signal: in forward direction - Error signal: in backward direction Learning in MLP Objective function where The desired output of the jth output neuron The real output of the jth output neuron Sum squared error function Steepest descent search method Partial derivative extension Learning rate parameter Synaptic weight from ith neuron in k-1th layer to the jth neuron in kth layer of the network Situation for Situation for Back propagation learning algorithm of MLP Updating equation where which is Back propagation formula For sigmoidal function f(.) we have Speeding the learning process Learning rate parameter Momentum constant parameter Heuristics for making the back-propagation algorithm perform better Sequential versus batch update Comparison, sequential model is Computationally faster More suitable for large and highly redundant training data set Makes the search in weight space stochastic in nature Less likely to be trapped in a local minimum More difficult to establish theoretical conditions for convergence of the algorithm Stopping criteria the back-propagation algorithm is considered to have converged gradient vector when the Euclidean norm of the gradient vector reaches a sufficiently small gradient threshold Squired error When the absolute rate of change in the average squared error per epoch is sufficiently small Generalization When generalization performance reaches a peak Generalization performance Overfitting downfitting generalization performance Cross-validation Leave-one-out Practical Considerations for Back-Propagation Learning How Many Hidden Units? Different Learning Rates for Different Layers? Overview We started by revisiting the concept of linear separability and the need for multi-layered neural networks We then saw how the Back-Prop