数字语音处理概论口语对话

数字语音处理概论口语对话1SpokenDialogueSystemsAlmostallhuman-networkinteractionscanbemadebyspokendialogueSpeechunderstanding,speechsynthesis,dialoguemanagement,discourseanalysisSystem/user/mixedinitiativesReliability/efficiency,dialoguemodeling/flowcontrolTransactionsuccessrate/averagedialogueturnsDatabasesSentenceGenerationandSpeechSynthesisOutputSpeechInputSpeechDialogueManagerSpeechRecognitionandUnderstandingUser’sIntentionDiscourseAnalysisResponsetotheuserInternetNetworksUsersDialogueServer2KeyProcessesinASpokenDialogueABasicFormulationgoal:thesystemtakestherightactionsaftereachdialogueturnandcompletethetasksuccessfullyfinally

ThreeKeyElementsspeechrecognitionandunderstanding:convertingXntosomesemanticinterpretationFndiscourseanalysis:convertingSn-1toSn,thenewdiscoursesemantics(dialoguestate),givenallpossibleFn

dialoguemanagement:selectthemostsuitableactionAngiventhediscoursesemantics(dialoguestate)SnXn:speechinputfromtheuserinthen-thdialogueturnSn:discoursesemantics(dialoguestate)atthen-thdialogueturnAn:action(response,actions,etc.)ofthesystem(computer,hand-helddevice,networkserver,etc.)afterthen-thdialogueturnbydialoguemanagementbydiscourseanalysisbyspeechrecognitionandunderstandingFn:semanticinterpretationoftheinputspeechXn3DialogueStructureTurnsanuninterruptedstreamofspeech(oneorseveralutterances/sentences)fromoneparticipantinadialoguespeakingturn:conveysnewinformation back-channelturn:acknowledgementandsoon(e.g.O.K.)Initiative-ResponsePairaturnmayincludebotharesponseandaninitiativesysteminitiative:thesystemalwaysleadstheinteractionflow userinitiative:theuserdecideshowtoproceed mixedinitiative:bothacceptabletosomedegreeSpeechActs(DialogueActs)goalorintentioncarriedbythespeechregardlessofthedetailedlinguisticformforwardlookingactsconversationopening(e.g.MayIhelpyou?),offer(e.g.TherearethreeflightstoTaipei…),assert(e.g.I’llleaveonTuesday),reassert(e.g.No,IsaidTuesday),informationrequest(e.g.Whendoesitdepart?),etc.backwardlookingactsaccept(e.g.Yes),accept-part(e.g.O.K.,buteconomyclass),reject(e.g.No),signalnotclear(e.g.Whatdidyousay?),etc.speechactslinguisticforms:amany-to-manymappinge.g.“O.K.”requestforconfirmation,confirmationtaskdependent/independenthelpfulinanalysis,modeling,training,systemdesign,etc.Sub-dialoguese.g.“askingfordestination”,“askingfordeparturetime”,…..4LanguageUnderstandingforLimitedDomainSemanticFrames

AnExampleforSemanticRepresentationasemanticclassdefinedbyanentityandanumberofattributes(orslots) e.g.[Flight]: [Airline](United) [Origin](SanFrancisco) [Destination](Boston) [Date](May18) [FlightNo](2306)“slot-and-filler”structureSentenceParsingwithContext-freeGrammar(CFG)forLanguageUnderstandingextensiontoProbabilisticCFG,integrationwithN-gram(localrelationwithoutsemantics),etc.Grammar(RewriteRules) SNPVP NPN VPV-clusterPP V-cluster(wouldliketo)V Vfly|go PPPrepNP NBoston|I PreptoSNPNV-clusterPPVPVNPPrepIwouldliketoflytoBostonN(wouldliketo)5RobustParsingforSpeechUnderstandingProblemsforSentenceParsingwithCFGungrammaticalutterancesspeechrecognitionerrors(substitutions,deletions,insertions)spontaneousspeechproblems:um–,cough,hesitation,repetition,repair,etc.unnecessarydetails,irrelevantwords,greetings,unlimitednumberoflinguisticformsforagivenact e.g.toBoston I’mgoingtoBoston,IneedbetoatBostonTomorrow um–justaminute–Iwishto–Iwishto–gotoBostonRobustParsingasanExampleApproachsmallgrammarsforparticularitemsinaverylimiteddomain,othershandledasfillers e.g.Destination→PrepCityName Prep→to|for|at CityName→Boston|LosAngeles|...differentsmallgrammarsmayoperatesimultaneouslykeywordspottinghelpfulconceptN-grammaybehelpful SpeechUnderstandingtwo-stage:speechrecognition(orkeywordspotting)followedbysemanticparsing(e.g.robustparsing)single-stage:integratedintoasinglestageCityName(Boston,...)direction(to,for...)similartoclass-basedN-gramProb(ci|ci-1),ci:concept6ConditionalRandomField(CRF)

ObservedvariablesTargetvariables7

ObservedvariablesTargetvariables

ConditionalRandomField(CRF)8ExamplePOSTaggingInputsequence:naturallanguagesentenceEx:“AmyatelunchatKFC”Outputsequence:POStaggingPossiblePOStagging:NOUN,VERB,ADJECTIVE,ADVERB,PREPOSITION…Ex:“Amy(NOUN)ate(VERB)lunch(NOUN)at(PREPOSITION)KFC(NOUN)”9POSTaggingPOSiisdeterminedbythewordiandPOSi-1

AmyatelunchatKFCInputwordoutputPOS1POS2POS3POS4POS5Example10Training/TestingofCRF

11Semi-conditionalRandomField(Semi-CRF)

12SlotfillingInputsequence:naturallanguagesentenceEx:FunnymovieaboutbridesmaidstarringKeiraKnightleyOutputsequence:slotsequenceGENRE,PLOT,ACTOREx:[Funny](GENRE)movieabout[bridesmaid](PLOT)starring[KeiraKnightley](ACTOR)Example13DiscourseAnalysisandDialogueManagementDiscourseAnalysisconversionfromrelativeexpressions(e.g.tomorrow,nextweek,he,it…)torealobjectsautomaticinference:decidingonmissinginformationbasedonavailableknowledge(e.g.“howmanyflightsinthemorning?”impliesthedestination/originpreviouslymentioned)inconsistency/ambiguitydetection(e.g.needclarificationbyconfirmation)exampleapproach:maintaining/updatingthedialoguestates(orsemanticslots)DialogueManagementcontrollingthedialogueflow,interactingwiththeuser,generatingthenextactione.g.askingforincompleteinformation,confirmation,clarifyinconsistency,fillinguptheemptyslotsone-by-onetowardsthecompletionofthetask,optimizingtheaccuracy/efficiency/userfriendlinessofthedialoguedialoguegrammar:finitestatemachinesasanexampleplan-baseddialoguemanagementasanotherexamplechallengingformixed-initiativedialoguesPerformanceMeasureinternal:worderrorrate,slotaccuracy(forunderstanding),etc.overall:averagesuccessrate(foraccuracy),averagenumberofturns(forefficiency),etc.Subdialogue:ConversationOpeningSubdialogue:AskingforDestinationSubdialogue:AskingforDepartureTimeDestination

filledupDepartureTimefilledupnoyesnoyes14DialogueManagementExampleApproach–MDP-basedExampleTask:flightbookingTheinformationthesystemneedstoknow:ThedeparturecityThearrivalcityDefinethestateas(DEPARTURE,ARRIVAL)Therearetotallyfourstates:(?,?),(KNOWN,?),(?,KNOWN),(KNOWN,KNOWN)15FlightBookingwithMDP(1/5)Thestateisdecidedbytheinformationthesystemknows.S1SfS2(?,?)(KNOWN,?)(KNOWN,KNOWN)16Thestateisdecidedbytheinformationthesystemknows.Asetofavailableactionsisalsodefined.S1SfS2A1:askDEPARTUREcityA2:askARRIVALcityA3:confirmA4:returnflightlistFlightBookingwithMDP(1/5)17AssumethesystemisatstateS1andtakesactionA1.S1SfS2A1:askDEPARTUREcityA1(?,?)FlightBookingwithMDP(2/5)18AssumethesystemisatstateS1andtakesactionA1.Userresponsewillcausethestatetotransit.S1SfS2A1:askDEPARTUREcityA1(?,?)FlightBookingwithMDP(2/5)19Thetransitionisprobabilisticbasedonuserresponseandrecognitionresults(witherrors).S1SfS2A1:askDEPARTUREcityA1(?,?)0.70.20.1Response:FromTaipei.Response:FromTaipeitoBoston.Response:Whatdidyousay?FlightBookingwithMDP(3/5)20Thetransitionisprobabilisticbasedonuserresponseandrecognitionresults(witherrors).Arewardassociatedwitheachtransition.S1SfS2A1:askDEPARTUREcityA1(?,?)0.70.20.1+10+5-5FlightBookingwithMDP(3/5)21Theinteractioncontinues.S1SfS2A2(KNOWN,?)A2:askARRIVALcityFlightBookingwithMDP(4/5)22Theinteractioncontinues.S1SfS2A2(KNOWN,?)A2:askARRIVALcityFlightBookingwithMDP(4/5)23Theinteractioncontinues.Whenthefinalstateisreached,thetaskiscompletedandresultisreturned.S1SfS2(KNOWN,KNOWN)FlightBookingwithMDP(4/5)24Fortheoveralldialoguesession,thegoalistomaximizethetotalreward

R=R1+…+Rn=5+5Dialogueoptimizedbychoosingarightactiongiveneachstate(policy).LearnedbyReinforcementLearning.ImprovedasPartiallyObservableMDP(POMDP)S1SfS2A1(?,?)+5A2+5FlightBookingwithMDP(5/5)25Client-ServerArchitectureGalaxy,MITIntegrationPlatform,AT&TDomainDependent/IndependentServersSharedbyDifferentApplications/Clientsreducingcomputationrequirementsatuser(client)byallocatingmostloadatserverhigherportabilitytodifferenttasksApplication(Client)Application(Client)Application(Client)D