lecture线性模型的扩展机器学习与数据科学

MachineLearningandData 200

f=x,

3 4–不可微

wTx=f

= v>-1v£5

y=f w

=fwTx=wTf6y=f

=wT其中f

7 定义代价函数为均方误差ew)= e2n) nh

2Nn

Dw=-Dw=-ew1=-N()Nen¶e8 最小均方 ：LeastMean其中e为时刻new=其中e为时刻n2 nh=- =-e ew=- =-e

9

基 n+

ew

ek2k˛ek

2n nw

w

n)=-

¶en

k()

eNe

¶

w)=1k2Nn=1k˛k

2nLL

w)=12Ln=1k˛

2nekek 算法:基本概:

t exp

t=a

eat-e-ateat+e-at

Rumelhart,Hinton&Rumelhart,Hinton&Williams,“Learningrepresentationsbyback-propagatingerrors”,Nature,1986j j

en

1e2n)

2j˛

av=

en=

ejn=1j˛

n)L Le=1e(n)=1e2n)j j

2Ln=1j˛

n

n=

n

n=-

ej

n

¶wji)

ej=dj-yjj–如果神经元jj

en

1e2n)

n=n)–C:

2j˛

N

–对所有n)诱导局部域：v)

mwy ji vj

n+Dwjie/¶wji

en=dn-

n)

n=n)–神经元j是输出节

1

“可见”误差的瞬时能量：e =nyiyj=fjjDwji)e/¶wji

n) en)en)ejnyjn)vjn)n)nynvjwjin)nvjyin)

n+=- =/=- =en=即

enenyn)vn)

enf

n)

n)n)nynvnwn)nn+hejjvjyi 神经元局部梯度的定义：–对于输出层神经元

n=en)=-n=-en)en=- ¶v

n

n)=

ejnfjv

n y

n=hejjvjyin=djnyi神经元局部梯度的定义：–

n=en)=-n=-en)en=-¶v

n

n)=

ejnfjv

n y

–n=-en)en)ejyjj

ynvj fjfj –对于隐藏层神经元jdjn=-

en)=-

enyn)=-

envnn)

nv

n)–其中en

n,

=

y=fvm–于m

22

vk=wkjyjn)jen)=

k

nykvk

jjkj¶yjjkj

¶y

k()

¶ekkkjdkjn)¶=-ej

=

n=-en)nj =-jvjnjnjvjnj j)

d=ef

n=-en)

en)

f

nen)j j

fj

ej=dkj

ej=dkj

n

dj)=ejnjvjDwjin=hejjvjyin)=hdjyij：ejdkjdj=ejnjj

n=hejvjyin)=hdjyi验证：Dwjihe/¶wjien)nwjiDwji=hdjn)yi权值校 局部梯 jvjnejjvjjjvje

yj=fjvj

=nyin)– – ek=ek=dk-ykdj=jvjnejhn)yi神经元j在隐藏层 dj=jvjej

t= exp-

t=aftftjvj=afvjfvj=ayj-yj若jyjojdj=jvjnej=adj-ojojn1-ojdjnf

n)-j

t=a

eat-e-ateat+

at

t=a1ftftdj=jvjnejdjnf

n)

xi,dimmljvljvn=nyln)j

yn=

vn

y0n

n

yLn

n)

ej=dj-ojnfvLn

j˛output

n=¢

fjv

j˛hidden

n1n+hnyln)

小 机器学习与数achineLearning&DataScience

ej=dj-yj1

n=n)

en

n) N

e=1

m

=nyin)yj=fjvj

wji‹Dwji)/ 称函数，如双曲正切，DropOut(G.E.Hinton,JMLRDropConnect(L.Wanetal.ICML

DropOut(G.E.Hinton,DropConnect(L.Wanetal.784-800x800x10的31.4%1.6%BP1.4%1.6%BP(Random1.2%1.2%RBM(784-BatchWithout[1]SergeyIoffe,ChristianSzegedy,"Batchnormalization:Acceleratingdeepnetworktrainingbyreducinginternalcovariateshift".ICML,2015.

XavierweightStochasticgradientdecentMomentumNesterovacceleratedgradient[1]XavierGlorot&YoshuaBengio,"Understandingthedifficultyoftrainingdeepfeed-forwardneuralnetworks",AI&STATS2010.DwjiaDwjihdji

则是什么？最优权值下，网络实际输出值是期望响应向量的条件期望的 ykj表示网络响应于xj的第k个神经元的输出：ykjFkxjTxT 2, M, j , ,..., TFx, 2, M, jN2 N2

R=12Nj

dj-Fxj

,jj如果Fk>Fj)"j„k则把输入向量x分类为 ，其Fk和Fj=1x,x,...,

[1]Richard&Lippmann:NeuralNetworkclassifiersestimateBayesianaposterioriprobabilities,Computation. [1]WebbA.R.andLoweD.:Theoptimalinternalrepresentationofmultilayerclassifiernetworksperformsnonlineardiscriminant ysis,[1]WebbA.R.andLoweD.:Theoptimalinternalrepresentationofmultilayerclassifiernetworksperformsnonlineardiscriminant ysis,NeuralNetworks,vol.5,pp.480-488,1990.–线性判别分析：假设有2个类别C1 »在分类意义上， Q/PerceptronGaussianGaussianModelofMixtureofExperts混合专家HierarchicalTopicmodel中，LatentDirichletHierarchicalSparse在KernelmachineDeepLearning

SupervisedSupervisedLearningofSemanticClassesforImageAnnotationandRetrieval,PAMI2007Q/法 理论上，一个隐藏层就足够作为通用函数近器地推广,因为根据理论,我们需要控制两个因素:经验ReLU(x)=max(x,[1]Krizhevsky,A.,Sutskever,I.,andHinton,G.E.Imagenetclassificationwithdeepconvolutionalneuralnetworks.InNIPS,2012. 个 MatthewD.ZeilerandRobFergus,"VisualizingandUnderstandingConvolutional 个 Stride2:相邻窗口每次滑动2个像素(或单位),i.e.5个像素 Filtersize7:使用7x7的kernel作用在每个窗口内，输出一个响应值3x3maxpooling:下采样(Downsampling)步骤在一个3x3邻域内的9个响应3x3maxpoolingstride2:相邻的3x3窗口每次滑动2个像素， 个 输入为 110x11055x55:使用3x3maxpoolingwithstride2396通道选用96个7x7x3的filters对224x224x3filter给出一个特征图(feature个 得到256个26x26的featuremaps,然后MaxPooling之后变成13x13x256第6层和第7层为全连接层(FullConnected

[1]LeCun,Yann,etal."Backpropagationappliedtohandwrittenzipcoderecognition."Neuralcomputation,1989.[2]LeCun,Yann,etal."Gradient-basedlearningappliedtorecognition."ProceedingsoftheIEEE(1998):2278-2324.[3]AlexKrizhevsky,IlyaSutskever,andGeoffreyE.Hinton."Imagenetclassificationwithdeepconvolutionalneuralnetworks."NIPS.2012.He,K.,Ren,S.,Sun,J.,&Zhang,X.:DeepResidualLearningforImageRecognition,CVPR2016. xyxyHe,K.,Ren,S.,Sun,J.,&Zhang,X.:DeepResidualLearningforImageRecognition,CVPR[1]Hinton,G.E.,Krizhevsky,A.,Srivastava,N.,Sutskever,I.,&Salakhutdinov,R.:Dropout:asimplewaytopreventneuralnetworksfromoverfitting,JournalofMachineLearningResearch,15,1929-1958,2014.[2]He,K.,Ren,S.,Sun,J.,&Zhang,X.:DeepResidualLearningforImageRecognition,arXiv,2016.[3]SergeyIoffe,ChristianSzegedy,"Batchnormalization:Acceleratingdeepnetworktrainingbyreducinginternalcovariateshift".ICML,2015.[4]Fei-Fei,L.,Karpathy,A.,Leung,T.,Shetty,S.,Sukthankar,R.,&Toderici,G.(2014):Large-Scale NeuralNetworks.IEEECVPR.[5