语言处理与潜在主题分析

14.0LinguisticProcessingandLatentTopicAnalysis

语言处理与潜在主题分析1

数位语音处理概论IntroductiontoDigitalSpeechProcessingDocumentsWordsTopicTkLatentSemanticAnalysis(LSA)2LatentSemanticAnalysis(LSA)-Word-DocumentMatrixRepresentationVocabularyVofsizeMandCorpusTofsizeNV={w1,w2,...wi,..wM},wi:thei-thword,e.g.M=2×104

T={d1,d2,...dj,..dN},dj:thej-thdocument,e.g.N=105cij:numberoftimeswioccursindj

nj:totalnumberofwordspresentindj

ti=Σj

cij:totalnumberoftimeswioccursinT

Word-DocumentMatrixW

W=[wij]eachrowofWisaN-dim“featurevector”forawordwiwithrespecttoalldocumentsdj eachcolumnofWisaM-dim“featurevector”foradocumentdjwithrespecttoallwordswid1d2........dj..........dNw1w2.wiwMwij3LatentSemanticAnalysis(LSA)j4DimensionalityReduction(1/2)

dimensionalityreduction:selectionofRlargesteigenvalues(R=800forexample)T2R“concepts”or“latentsemanticconcepts”5DimensionalityReduction(2/2)

dimensionalityreduction:selectionofRlargesteigenvalues2TTsi2

:weights(significanceofthe“componentmatrices”e′ie′iT)R“concepts”or“latentsemanticconcepts”(i,j)elementofWTW:innerproductofi-thandj-thcolumnsofW

“similarity”betweendianddjTNN6SingularValueDecomposition(SVD)

SingularValueDecomposition(SVD)

si:singularvalues,s1≥s2....≥sR U:leftsingularmatrix,V:rightsingularmatrixVectorsforwordwi:uiS=ui(arow)avectorwithdimensionalityNreducedtoavectoruiS=uiwithdimensionalityRN-dimensionalspacedefinedbyNdocumentsreducedtoR-dimensionalspacedefinedbyR“concepts”theRrowvectorsofVT,orcolumnvectorsofV,oreigenvectors{e′1,..e′R},aretheRorthonormalbasisforthe“latentsemanticspace”withdimensionalityR,withwhichuiS=u

iisrepresentedwordswithsimilar“semanticconcepts”have“closer”locationinthe“latentsemanticspace”theytendtoappearinsimilar“types”ofdocuments,althoughnotnecessarilyinexactlythesamedocumentsd1d2........dj..........dNw1w2wiwMwij=w1w2wiwMuiUR×Rs1sRd1d2........dj..........dNVTR×NM×RvjT7SingularValueDecomposition(SVD)dp=USvpT(justasacolumninW=USVT)8SingularValueDecomposition(SVD)

SingularValueDecomposition(SVD)

Vectorsfordocumentdj:vjS=vj(arow,orvj=SvjTforacolumn)avectorwithdimentionalityMreducedtoavectorvjS=vjwithdimentionalityRM-dimentionalspacedefinedbyMwordsreducedtoR-dimentionalspacedefinedbyR“concepts”theRcolumnsofU,oreigenvectors{e1,...eR},aretheRorthonormalbasisforthe“latentsemanticspace”withdimensionalityR,withwhichvjS=vjisrepresenteddocumentswithsimilar“semanticconcepts”have“closer”locationinthe“latentsemanticspace”theytendtoincludesimilar“types”ofwords,althoughnotnecessarilyexactlythesamewordsTheAssociationStructurebetweenwordswianddocuments

djispreservedwithnoisyinformationdeleted,whilethedimensionalityisreducedtoacommonsetofR“concepts”d1d2........dj..........dNw1w2wiwMwij=w1w2wiwMuiUR×Rs1sRd1d2........dj..........dNvjVTR×NM×RTT9ExampleApplicationsinLinguisticProcessing

WordClustering

exampleapplications:class-basedlanguagemodeling,informationretrieval,etc.wordswithsimilar“semanticconcepts”have“closer”locationinthe“latentsemanticspace”theytendtoappearinsimilar“types”ofdocuments,althoughnotnecessarilyinexactlythesamedocumentseachcomponentinthereducedwordvectorujS=ujisthe“association”ofthewordwiththecorresponding“concept”examplesimilaritymeasurebetweentwowords:DocumentClusteringexampleapplications:clusteredlanguagemodeling,languagemodeladaptation,informationretrieval,etc.documentswithsimilar“semanticconcepts”have“closer”locationinthe“latentsemanticspace”theytendtoincludesimilar“types”ofwords,althoughnotnecessarilyexactlythesamewordseachcomponentonthereduceddocumentvectorvjS=vjisthe“association”ofthedocumentwiththecorresponding“concept”example“similarity”measurebetweentwodocuments:2210LSAforLinguisticProcessing

CosineSimilarity

magnitudeSimilarity

11ExampleApplicationsinLinguisticProcessingInformationRetrieval“conceptmatching”vs“lexicalmatching”:relevantdocumentsareassociatedwithsimilar“concepts”,butmaynotincludeexactlythesamewordsexampleapproach:treatingthequeryasanewdocument(by“folding-in”),andevaluatingits“similarity”withallpossibledocumentsFold-in

consideranewdocumentoutsideofthetrainingcorpusT,butwithsimilarlanguagepatternsor“concepts”constructanewcolumndp,p>N,withrespecttotheMwordsassumingUandSremainunchanged dp=USvpT(justasacolumninW=USVT)

v

p=vpS=dpTUasanR-dimrepresentationofthenewdocument (i.e.obtainingtheprojectionofdponthebasiseiofUbyinnerproduct)12IntegrationwithN-gramLanguageModelsLanguageModelingforSpeechRecognitionProb(wq|dq-1) wq:theq-thwordinthecurrentdocumenttoberecognized(q:sequenceindex) dq-1:therecognizedhistoryinthecurrentdocument

v

q-1=dq-1TU:representationofdq-1byvq-1(folded-in)Prob(wq|dq-1)canbeestimatedbyuqand

v

q-1intheR-dimspaceintegrationwithN-gram Prob(wq|Hq-1)=Prob(wq|hq-1,

dq-1) Hq-1:historyuptowq-1

hq-1:<wq-n+1,wq-n+2,...wq-1>N-gramgiveslocalrelationships,whiledq-1givessemanticconceptsdq-1emphasizesmorethekeycontentwords,whileN-gramcountsallwordssimilarlyincludingfunctionwordsv

q-1fordq-1canbeestimatediterativelyassumingtheq-thwordinthecurrentdocumentiswi(n)(n)v

qmovesintheR-dimspaceinitially,eventuallysettledownsomewherei-thdimensionalityoutofMT13ProbabilisticLatentSemanticAnalysis(PLSA)ExactlythesameasLSA,usingasetoflatenttopics{}toconstructanewrelationshipbetweenthedocumentsandterms,butwithaprobabilisticframeworkTrainedwithEMbymaximizingthetotallikelihood

:frequencycountofterminthedocumentDi:

documentsTk:latenttopicstj:terms14ProbabilisticLatentSemanticAnalysis(PLSA)w:wordz:topicd:documentN:wordsindocumentdM:documentsincorpus

15

(k:topicindex,atotalofKtopics)

LatentDirichletAllocation(LDA)AdocumentisrepresentedasrandommixturesoflatenttopicsEachtopicischaracterizedbyadistributionoverwords16GibbsSamplingingeneral

17GibbsSamplingappliedonLDA…w11?w12?w13?w1n?…Doc1w21?w22?w23?w2n?…Doc2SampleP(Z,W):TopicWord18GibbsSamplingappliedonLDA…w11w12w13w1n…Doc1w21w22w23w2n…Doc2SampleP(Z,W):RandomInitialization19GibbsSamplingappliedonLDA…w11?w12w13w1n…Doc1w21w22w23w2n…Doc2SampleP(Z,W):RandomInitializationEraseZ11,anddrawanewZ11~

20GibbsSamplingappliedonLDA…w11w12?w13w1n…Doc1w21w22w23w2n…Doc2SampleP(Z,W):RandomInitializationEraseZ11,anddrawanewZ11~EraseZ12,anddrawanewZ12~

21GibbsSamplingappliedonLDAw11w12w13w1nw21w22w23w2n………Doc1Doc2SampleP(Z,W):RandomInitializationEraseZ11,anddrawanewZ11~EraseZ12,anddrawanewZ12~IterativelyupdatetopicassignmentforeachworduntilconvergeComputeθ,φaccordingtothefinalsetting

22MatrixFactorization(MF)forRecommendationsystemsMovie1Movie2Movie3Movie4Movie5Movie6Movie7Movie8Movie9UserA3.74.0UserB4.04.3UserC4.1UserD2.32.5UserE3.3UserF2.9UserG2.62.7

23MatrixFactorization(MF)Mappingbothusersanditemstoajointlatentfactorspaceofdimensionalityf

iI11uU=1uUiI1fflatentfactor:towardsmale,seriousness,etc.24MatrixFactorization(MF)

25OverfittingProblemAgoodmodelisnotjusttofitallthetrainingdataneedstocoverunseendatawellwhichmayhavedistributionsslightlydifferentfromthatoftrainingdatatoocomplicatedmodelswithtoomanyparametersusuallyleadstooverfitting26ExtensionsofMatrixFactorization(MF)BiasedMFaddglobalbiasμ

(usually=averagerating)

,userbias

bu

,anditembiasbi

asparametersNon-negativeMatrixFactorizationrestricttheva