口语文档理解与组织以促进用户内容交互

11.0SpokenDocumentUnderstandingandOrganizationforUser-contentInteraction口语文档理解与组织以促进用户内容交互1

数字语音处理概论IntroductiontoDigitalSpeechProcessingUser-ContentInteractionforSpokenContentRetrievalProblemsUnliketextcontent,spokencontentnoteasilysummarizedonscreen,thusretrievedresultsdifficulttoscanandselectUser-contentinteractionalwaysimportantevenfortextcontentPossibleApproachesAutomaticsummary/titlegenerationandkeytermextractionforspokencontentSemanticstructuringforspokencontentMulti-modaldialoguewithimprovedinteractionKeyTerms/Titles/SummariesUserQueryMulti-modalDialogueSpokenArchivesRetrievedResultsRetrievalEngineUserInterfaceSemanticStructuring2Multi-media/SpokenDocumentUnderstandingandOrganizationKeyTerm/NamedEntityExtractionfromMulti-media/SpokenDocuments

—personalnames,organizationnames,locationnames,eventnames —keyphrase/keywordsinthedocuments —veryoftenout-of-vocabulary(OOV)words,difficultforrecognitionMulti-media/SpokenDocumentSegmentation—automaticallysegmentingamulti-media/spokendocumentintoshortparagraphs,eachwithacentraltopicInformationExtractionforMulti-media/SpokenDocuments—extractionofkeyinformationsuchaswho,when,where,whatandhowfortheinformationdescribedbymulti-media/spokendocuments.—veryoftentherelationshipsamongthekeyterms/namedentitiesSummarizationforMulti-media/SpokenDocuments—automaticallygeneratingasummary(intextorspeechform)foreachshortparagraphTitleGenerationforMulti-media/SpokenDocuments—automaticallygeneratingatitle(intextorspeechform)foreachshortparagraph—veryconcisesummaryindicatingthetopicareaTopicAnalysisandOrganizationforMulti-media/SpokenDocuments—analyzingthesubjecttopicsfortheshortparagraphs—clusteringandorganizingthesubjecttopicsoftheshortparagraphs,givingtherelationshipsamongthemforeasieraccess3IntegrationRelationshipsamongtheInvolvedTechnologyAreasKeyterms/NamedEntityExtractionfromSpokenDocumentsSemanticAnalysisInformationIndexing,RetrievalAndBrowsingKeyTermExtractionfromSpokenDocuments4KeyTermExtractionfromSpokenContent(1/2)KeyTerms:keyphrasesandkeywordsKeyPhraseBoundaryDetectionAnExampleLeft/rightboundaryofakeyphrasedetectedbycontextstatistics“hidden”almostalwaysfollowedbythesameword“hiddenMarkov”almostalwaysfollowedbythesameword“hiddenMarkovmodel”isfollowedbymanydifferentwordsboundaryhiddenMarkovmodelrepresentiscan::isofin::5KeyTermExtractionfromSpokenContent(2/2)ProsodicFeatureskeytermsprobablyproducedwithlongerduration,widerpitchrangeandhigherenergySemanticFeatures(e.g.PLSA)keytermsusuallyfocusedonsmallernumberoftopicsLexicalFeaturesTF/IDF,POStag,etc.NotkeytermP(Tk|ti)kkeytermP(Tk|ti)ktopicstopics6X1X2X3X4X5X6t2t1documentd:Correctlyrecognizedwordsummaryofdocumentd:SelectingmostrepresentativeutterancesintheoriginaldocumentbutavoidingredundancyX1X3Wronglyrecognizedword-Scoringsentencesbasedonprosodic,semantic,lexicalfeaturesandconfidencemeasures,etc.-BasedonagivensummarizationratioExtractiveSummarizationofSpokenDocuments7Titlesforretrieveddocuments/segmentshelpfulinbrowsingandselectionofretrievedresultsShort,readable,tellingwhatthedocument/segmentisaboutOneexample:ScoredViterbiSearchTitleGenerationforSpokenDocumentsTrainingcorpusTermOrderingModelTermSelectionModelTitleLengthModelSpokendocumentRecognitionandSummarizationViterbiAlgorithmOutputTitleSummary8Example1:retrievedresultsclusteredbyLatentTopicsandorganizedinatwo-dimensionaltreestructure(multi-layeredmap)eachclusterlabeledbyasetofkeytermsrepresentingagroupofretrieveddocuments/segmentseachclusterexpandedintoamapinthenextlayerSemanticStructuring(1/2)9Example2:Key-termGrapheachretrievedspokendocument/segmentlabeledbyasetofkeytermsrelationshipsbetweenkeytermsrepresentedbyagraphSemanticStructuring(2/2)---------------------------------------------------------------------------retrievedspokendocumentskeytermgraphAcousticModelingViterbisearchHMMLanguageModelingPerplexity10Anexample:user-systeminteractionmodeledasaMarkovDecisionProcess(MDP)Multi-modalDialogueKeyTerms/Titles/SummariesSpokenArchivesUserRetrievedResultsRetrievalEngineQueryUserInterfaceMulti-modalDialogueSemanticStructuringExamplegoalssmallaveragenumberofdialogueturns(averagenumberofuseractionstaken)forsuccessfultasks(success:user’sinformationneedsatisfied)lesseffortforuser,betterretrievalquality11SpokenDocumentSummarizationWhysummarization?HugequantitiesofinformationSpokencontentdifficulttobeshownonthescreenanddifficulttobrowseNews

articlesWebsitesSocial

MediaBooksMailsBroadcastNewsMeetingLecture12SpokenDocumentSummarizationMoredifficultthantextsummarizationRecognitionerrors,Disfluency,etc.ExtrainformationnotintextProsody,speakeridentity,emotion,etc.ASRSystemSummarization

System

dN:document

d2:document

d1:document

.SN:Summary

S2:Summary

S1:Summary

.....AudioRecording13UnsupervisedApproach:MaximumMarginRelevance(MMR)

PresentlySelectedSummaryS

…………

…………

14SupervisedApproach:SVMorSimilarSN:SummaryS2:SummarydN:documentd2:documentd1:document

...S1:Summary

...HumanlabeledTrainingdataBinaryClassificationmodelFeatureExtraction

BinaryClassificationmodelTrainingphaseTestingphaseRankedutterances

FeatureExtractionASRSystemTestingdata

Trainedwithdocumentswithhumanlabeledsummaries15DomainAdaptationofSupervisedApproachProblemHardtogethighqualitytrainingdataInmostcases,wehavelabeledout-of-domainreferencesbutnotlabeledtargetdomainreferencesGoalTakingadvantageofout-of-domaindataOut-of-domain(News)TargetDomain(Lecture)?16DomainAdaptationofSupervisedApproach

SN:SummaryS2:SummarydN:documentd2:documentd1:document

...S1:Summary...HumanlabeledSpokenDocumentSummarymodeltraining

...

SummaryExtractionOut-of-domaindatawithlabeleddocument/summaryTargetdomaindatawithoutlabeleddocument/summary

17DomainAdaptationofSupervisedApproach

SN:SummaryS2:SummarydN:documentd2:documentd1:document

...S1:Summary...HumanlabeledSpokenDocumentSummarymodeltraining

...

SummaryExtractionOut-of-domaindatawithlabeleddocument/summaryTargetdomaindatawithoutlabeleddocument/summary

18DocumentSummarizationExtractiveSummarizationselectsentencesinthedocumentAbstractiveSummarizationGeneratesentencesdescribingthecontentofthedocumente.g.彰化检方侦办芳苑乡公所道路排水改善工程弊案拘提芳苑乡长陈聪明检方认为陈聪明等人和包商勾结涉嫌贪污和图利罪嫌凌晨向法院声请羁押以及公所秘书杨腾煌获准彰化

乡公所

陈聪明

涉嫌贪污彰化检方侦办芳苑乡公所道路排水改善工程弊案拘提芳苑乡长陈聪明ExtractiveAbstractiveSummarization

System19DocumentSummarizatione.g.彰化

检方侦办芳苑乡公所道路排水改善工程弊案拘提芳苑乡长陈聪明检方认为陈聪明等人和包商勾结涉嫌贪污和图利罪嫌凌晨向法院声请羁押以及公所秘书杨腾煌获准彰化

乡公所

陈聪明

涉嫌贪污

彰化检方侦办芳苑乡公所道路排水改善工程弊案拘提芳苑乡长陈聪明ExtractiveAbstractiveSummarization

SystemExtractiveSummarizationselectsentencesinthedocumentAbstractiveSummarizationGeneratesentencesdescribingthecontentofthedocument20AbstractiveSummarization(1/4)AnExampleApproachGeneratingcandidatesentencesbyagraphSelectingsentencesbytopicmodels,languagemodelsofwords,parts-of-speech(POS),lengthconstraint,etc.d1:document

1)GeneratingCandidatesentences2)SentenceselectionRankedlist

..…

……

21AbstractiveSummarization(2/4)X1:这个饭店房间算舒适.X2:这个饭店的房间很舒适但离市中心太远不方便X3:饭店挺漂亮但房间很旧X4:离市中心远1)GeneratingCandidatesentencesGraphconstruction+searchongraphNode:“word”inthesentenceEdge:wordorderinginthesentence22AbstractiveSummarization(3/4)X1:这个饭店房间算舒适X2:这个饭店的房间很舒适但离市中心太远不方便X3:饭店挺漂亮但房间很旧X4:离市中心远但离市中心太远不方便这个饭店房间算舒适漂亮挺很旧的1)GeneratingCandidatesentencesGraphconstruction+searchongraph23AbstractiveSummarization(3/4)X1:这个饭店房间算舒适X2:这个饭店的房间很舒适但离市中心太远不方便X3:饭店挺漂亮但房间很旧X4:离市中心远1)GeneratingCandidatesentencesGraphconstruction+searchongraph但离市中心太远不方便这个房间漂亮挺很旧的Startnode饭店算舒适24AbstractiveSummarization(3/4)X1:这个饭店房间算舒适X2:这个饭店的房间很舒适但离市中心太远不方便X3:饭店挺漂亮但房间很旧X4:离市中心远1)GeneratingCandidatesentencesGraphconstruction+searchongraph但离市中心太远不方便这个房间漂亮挺很旧的StartnodeEndnode饭店算舒适251)GenerateCandidatesentencesGraphconstruction+

searchongraphSearch

:

findValidpathongraphValidpath:pathfromstartnodetoendnodeX1:这个饭店房间算舒适X2:这个饭店的房间很舒适但离市中心太远不方便X3:饭店挺漂亮但房间很旧X4:离市中心远e.g.饭店房間很舒適但離市中心远但离市中心太远不方便这个房间漂亮挺很旧的StartnodeEndnode饭店算舒适AbstractiveSummarization(4/4)261)GeneratingCandidatesentencesGraphconstruction+searchongraphSearch

:

findValidpathongraphValidpath:pathfromstartnodetoendnodeAbstractiveSummarization(4/4)但离市中心太远不方便这个饭店房间算舒适漂亮挺很旧的StartnodeEndnodee.g.饭店房間很舒適但離市中心远饭店挺漂亮但房間很旧X1:这个饭店房间算舒适X2:这个饭店的房间很舒适但离市中心太远不方便X3:饭店挺漂亮但房间很旧X4:离市中心远27Interactivedialogue:retrievalengineinteractswiththeusertofindoutmorepreciselyhisinformationneedUserenteringthequeryWhentheretrievedresultsaredivergent,thesystemmayaskformoreinformationratherthanofferingtheresultsSpokenArchiveRetrievalEngineSystemresponseUSAPresidentMulti-modalInteractiveDialogueMorepreciselyplease?document305document116document298...Query128RetrievalEngineInternationalAffairsMulti-modalInteractiveDialogueInteractivedialogue:retrievalengineinteractswiththeusertofindoutmorepreciselyhisinformationneedUserenteringthesecondquerywhentheretrievedresultsarestilldivergent,butseemtohaveamajortrend,thesystemmayuseakeywordrepresentingthemajortrendaskingforconfirmationUsermayreply：“Yes”or“No,Asia”SystemresponseSpokenArchiveQuery2RegardingMiddleEast?document496document275document312...29MarkovDecisionProcess(MDP)Amathematicalframeworkfordecisionmaking,definedby(S,A,T,R,π)S:Setofstates,currentsystemstatusA:SetofactionsthesystemcantakeateachstateT:transitionprobabilitiesbetweenstateswhenacertainactionistakenR:rewardreceivedwhentakinganactionπ:policy,choiceofactiongiventhestateObjective:Findapolicythatmaximizestheexpectedtotalreward

30ModelasMarkovDecisionProcess(MDP)Afteraqueryentered,thesystemstartsatacertainstateStates:retrievalresultqualityestimatedasacontinuousvariable(e.g.MAP)plusthepresentdialogueturnAction:ateachstate,thereisasetofactionswhichcanbetaken:askingformoreinformation,returningakeywordoradocument,oralistofkeywordsordocumentsaskingforselectingone,Multi-modalInteractiveDialogueS1S2S3A1R1R2A2REndShowA2A3or

showingresults….Userresponsecorrespondstoacertainnegativereward(extraworkforuser)whenthesystemdecidestoshowtotheusertheretrievedresults,itearnssomepositivereward(e.g.MAPimprovement)Learnapolicymaximizingrewardsfromhistoricaluserinteractions(π:Si→Aj)31ReinforcementLearning

32Question-Answering(QA)inSpeechKnowledgeSourceQuestionAnsweringQuestionAnswerQuestion,Answer,KnowledgeSourcecanallbeintextformorinSpeechSpokenQuestionAnsweringbecomesimportantspokenquestionsandanswersareattractivetheavailabilityoflargenumberofon-linecoursesandsharedvideostodaymakesspokenanswersbydistinguishedinstructorsorspeakersmorefeasible,etc.TextKnowledgeSourceisalwaysimportant33ThreeTypesofQAFactoidQA:WhatisthenameofthelargestcityofTaiwan?Ans:Taipei.DefinitionalQA:WhatisQA?ComplexQuestion:HowtoconstructaQAsystem?34FactoidQAQuestionProcessingQueryFormulation:transformthequestionintoaqueryforretrievalAnswerTypeDetection(cityname,number,time,etc.)PassageRetrievalDocumentRetrieval,PassageRetrievalAnswerProcessingFindandrankcandidateanswers35FactoidQA–QuestionProcessingQueryFormulation:ChoosekeytermsfromthequestionEx:WhatisthenameofthelargestcityofTaiwan?“Taiwan”,“largestcity”arekeytermsandusedasqueryAnswerTypeDetection“cityname”forexampleLargenumberofhierarchicalclasseshand-craftedorautomaticallylearned36AnExampleFactoidQAWatson:aQAsystemdevelopbyIBM(text-based,nospeech),whowon“Jeopardy!”37DefinitionalQADefinitionalQA≈Query-focusedsummarizationUsesimilarframeworkasFactoidQAQuestionProcessingPassageRetrievalAnswerProcessingisreplacedbySummarization38ReferencesKeyterms“AutomaticKeyTermExtractionFromSpokenCourseLecturesUsingBranchingEntropyandProsodic/SemanticFeatures”,IEEEWorkshoponSpokenLanguageTechnology,Berkeley,California,U.S.A.,Dec2010,pp.253-258.“UnsupervisedTwo-StageKeywordExtractionfromSpokenDocumentsbyTopicCoherenceandSupportVectorMachine”,InternationalConferenceonAcoustics,SpeechandSignalProcessing,Kyoto,Japan,Mar2012,pp.5041-5044.TitleGeneration“AutomaticTitleGenerationforSpokenDocumentswithaDelicateScoredViterbiAlgorithm”,2ndIEEEWorkshoponSpokenLanguageTechnology,Goa,India,Dec2008,pp.165-168.39ReferencesSummarization“SupervisedSpokenDocumentSummarizationJointlyConsideringUtteranceImportanceandRedundancybyStructuredSupportVectorMachine”,Interspeech,Portland,U.S.A.,Sep2012.“UnsupervisedDomainAdaptationforSpokenDocumentSummarizationwithStructuredSupportVectorMachine”,InternationalConferenceonAcoustics,SpeechandSignalProcessing,Vancouver,Canada,May2013.“SemanticAnalysisandOrganizationofSpokenDocumentsBasedonParametersDerivedfromLatentTopics”,IEEETransactionsonAudio,SpeechandLanguageProcessing,Vol.19,No.7,Sep2011,pp.1875-1889."SpokenLectureSummarizationbyRando