深度学习框架比较-caffe的理念及使用课件

1、Introduction of Current Deep Learning Software Packages,1,学习交流PPT,Three Popular ones,1.Caffe / 2.Theano 3.TensorFlow These websites provide information about how to install and run related deep learning software.,2,学习交流PPT,1.Caffe,1.Overview: Caffe(Convolutional Architecture For Feature Extraction)

2、Created by Yangqing Jia(贾扬清), UC Berkeley. Written in C+, has Python and MATLAB interface. 2.Github page: 3.Tutorial:,4.Install method(CUDA+Caffe): Ouxinyu.github.io/Blogs/2014723001.html,3,学习交流PPT,Anatomy of Caffe, Blob: Stores data and derivatives Layer: Transforms Bottom blobs to top blobs Net: M

3、any layers; computes gradients via forward / backward,Blob,Layer,Net,4,学习交流PPT,Blob,A Blob is a wrapper over the actual data being processed and passed along by Caffe, and also under the hood provides synchronization capability between the CPU and the GPU.,The conventional blob dimensions for batche

4、s of image data are (number N) x (channel K) x (height H) x (width W).,For a convolution layer with 96 filters of 11 x 11 spatial dimension and 3 inputs the blob is 96 x 3 x 11 x 11. For an inner product / fully-connected layer with 1000 output channels and 1024 input channels the parameter blob is

5、1000 x 1024.,5,学习交流PPT,Layer,The layer is the essence of a model and the fundamental unit of computation. Layers convolve filters, pool, take inner products, apply nonlinearities like rectified-linear and sigmoid and other element-wise transformations, normalize, load data, and compute losses like s

6、oftmax and hinge.,6,学习交流PPT,Case：Convolution Layer,7,学习交流PPT,Net,The net jointly defines a function and its gradient by composition and auto-differentiation. The composition of every layers output computes the function to do a given task, and the composition of every layers backward computes the gra

7、dient from the loss to learn the task.,name: LogReg layer name: mnist type: Data top: data top: label data_param source: input_leveldb batch_size: 64 layer name: ip type: InnerProduct bottom: data top: ip inner_product_param num_output: 2 layer name: loss type: SoftmaxWithLoss bottom: ip bottom: lab

8、el top: loss ,8,学习交流PPT,How to use Caffe ?,Just 4 steps! 1. Convert data (run a script) 2. Define net (edit prototxt) 3. Define solver (edit prototxt) 4. Train (with pretrained weights) (run a script) Take Cifar10 image classification for example.,9,学习交流PPT, DataLayer reading from LMDB is the easies

9、t, create LMDB using convert_imageset Need text each line is “path/to/image.jpeg label”(use imageDataLayer read) Create HDF5 using h5py(use HDF5Layer read),Step 1: Convert Data for Caffe,10,学习交流PPT,Convert Data on CIFAR10,11,学习交流PPT,Step 2: Define Net(cifar10_quick_train_totxt),Layer name,Bl

10、obs name,Learning rate of weight,Learning rate of bias,Input image num per iteration,Training image data,Data type,Blobs name,12,学习交流PPT,Number of output class,Output accuracy during test,Output loss during train,If you finetune some pre-train model, you can set lr_mul=0,Step 2: Define Net(cifar10_q

11、uick_train_totxt),13,学习交流PPT,Visualize the Defined Network,14,学习交流PPT,Step 3: Define Solver (cifar10_quick_totxt),# reduce the learning rate after 8 epochs (4000 iters) by a factor of 10 # The train/test net protocol buffer definition net: examples/cifar10/cifar10_quick_train_test.

12、prototxt“ # test_iter specifies how many forward passes the test should carry out. # In the case of MNIST, we have test batch size 100 and 100 test iterations, # covering the full 10,000 testing images. test_iter: 100 # Carry out testing every 500 training iterations. test_interval: 500 # The base l

13、earning rate, momentum and the weight decay of the network. base_lr: 0.001 momentum: 0.9 weight_decay: 0.004 # The learning rate policylr_policy: fixed“ # Display every 100 iterations display: 100 # The maximum number of iterations max_iter: 4000 # snapshot intermediate results snapshot: 4000 snapsh

14、ot_prefix: examples/cifar10/cifar10_quick“ # solver mode: CPU or GPU solver_mode: GPU,Defined Net file,Key parameters,Important parameters,15,学习交流PPT,Step 4: Train,Write a shell ):,Then enjoy a cup of caffe,16,学习交流PPT,Model Zoo (Pre-trained Model + Finetune),We can finetune these models or do featur

15、e extraction based on these models,17,学习交流PPT,Some tricks/skills about training Caffe,1 Neural Networks: tricks of the trade,1.Data Augmentation to enlarge training samples2.Image Pre-Processing3.Network Initializations4.During Training5.Activation Functions6. Regularizationsmore details can refer t

16、o 1,2,2,18,学习交流PPT,Data Augmentation,19,学习交流PPT,Data Augmentation,Very useful for face and car recognition!,20,学习交流PPT,Data Augmentation,To get rid of occlusion and scale change, like visual tracking,21,学习交流PPT,Data Augmentation,22,学习交流PPT,Data Augmentation,23,学习交流PPT,Image Pre-Processing,Step1: sub

17、tract the dataset-mean value in each channel,Step2: s from RGB to BGR,Step3: move image channels to outermost dimension,Step4: rescale from 0, 1 to 0, 255,24,学习交流PPT,Network Initializations,25,学习交流PPT,During Training,help alleviate overfitting during training in Caffe,1 Srivastava, Nitish, et al. Dr

18、opout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15.1 (2014): 1929-1958. 2 S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift.arXiv preprint arXiv:1502.03167, 2015,Overfitting,26,学

19、习交流PPT,Pros and Cons of Caffe,27,学习交流PPT,A practical example of Caffe,Object detectionRCNN/Fast-RCNN/Faster-RCNN Caffe+MATLAB,28,学习交流PPT,lr = 0.1 x base learning rate,lr = base learning rate,29,学习交流PPT,30,学习交流PPT,31,学习交流PPT,32,学习交流PPT,2.Theano,1.Overview: A Python library that allows to define, opti

20、mize and evaluate mathematical expression. From Yoshua Bengios group at University of Montreal. Embracing computation graphs, symbolic computation. High-level wrappers: Keras, Lasagne. 2.Github: 3.Tutorial:,33,学习交流PPT,34,学习交流PPT,Pros and Cons of Theano,35,学习交流PPT,3.TensorFlow,1.Overview: Very similar to Theano - all about computation graphs. Easy visualizations (TensorBoard). Multi-G