计算机辅助语言学习CALL

12.0Computer-Assisted

Language

Learning

(CALL)计算机辅助语言学习CALL

数字语音处理概论IntroductiontoDigitalSpeechProcessing1References

for

12.0

“AnOverviewofSpokenLanguageTechnologyforEducation”,SpeechCommunications,51,pp.832-844,2009“Computer-assistedLanguageLearning(CALL)Systems”,Tutorial,Interspeech2012“ARecursiveDialogueGameforPersonalizedComputer-AidedPronunciationTraining”,IEEE/ACMTransactionsonAudio,SpeechandLanguageProcessing,Vol.23,No.1,Jan2015,pp.127-141.“SupervisedDetectionandUnsupervisedDiscoveryofPronunciationErrorPatternsforComputer-AssistedLanguageLearning”,IEEE/ACMTransactionsonAudio,SpeechandLanguageProcessing,Vol.23,No.3,Mar2015,pp.564-579.2Computer-AssistedLanguageLearning(CALL)GlobalizedWorldeveryoneneedstolearnoneormorelanguagesinadditiontothenativelanguageLanguageLearningone-to-onetutoringmosteffectivebutwithhighcostComputersnotasgoodasHumanTutorssoftwarereproducedeasilyusedrepeatedlyanytime,anywherenevergettiredorbored3ComponentsandsentencecompositionPhonemeset,Vocabulary,GrammarPronunciation:PhoneticandProsodicPhonemeWord+tones,stress,etc.Sentence+intonation,etc.Paragraph+prominence,etc.Computer-aidedPronunciationTraining(CAPT)ReadingWriting(Chinesecharacters,etc.)ListeningSpeakingDialoguesandCommunicationstravel/shopping,business/negotiation,etc.TargetSkillsofCALL4Learnersaresupposedtolearnhowtocontrolthearticulators(vocaltract)ButthemovementoftheseorgansisnoteasytoobserveObservationfromsignalsisfeasible,butnoteasytolearnbasedonsignalseitherFromArticulationtoSpeech5TalkingHeadshowingcorrectarticulationAcoustic-to-articulatoryinversiontoestimatethearticulatorymovementsVisualPresentationofArticulationStilldifficultforlearners6Computer-AidedPronunciationTraining(CAPT)QualitativeassessmentofpronunciationforlearnersErrorpatterndetection7Youmispronounced/th/as/s/CommonlyUsedApproachesIwearblueclose(cloth)todayYourpronunciationscoreis78.27Computer-AidedPronunciationTraining(CAPT)QualitativeassessmentofpronunciationforlearnersErrorpatterndetectionSpokenDialogueSystemImmersiveinteractiveenvironment8CommonlyUsedApproachesThanksIwearblueclose(cloth)todayYou’rewelcomeOh,Itlooksniceonyou!8Computer-AidedPronunciationTraining(CAPT)QualitativeassessmentofpronunciationforlearnersErrorpatterndetectionSpokenDialogueSystemImmersiveinteractiveenvironmentCorrectivefeedbackduringinteraction9CommonlyUsedApproachesDoyoumeanCLOTH?Iwearblueclose(cloth)todayOhyes,clothGreat9PronunciationScoringandErrorPatternDetectionProsodicModelsAcousticModelsPronunciationModelsSpeechSignalRepresentationAlignment/SegmentationErrorPatternDetectionScoring10AlignmentProblem–InsertionError11NocorrespondingsyllableinL1(ex.)sea→sheNocorrespondingphonemeinL1(ex.)r→l,v→bVowelinsertions(ex.)b-r→b-uh-rErrorPredictioninPronunciationModelingSEsbuhrlErrorehthuh12Native-likenessHowclosetogoldennativespeakers?whoarethe“golden”speakers?Modelstrainedwithagroupof“good”speakersIntelligibilityHowdistinguishable(lessconfusable)fromotherphonemes?LearningfromHumanLanguageTeachersTrainedtoofferscoresorerrorpatternsclosetoscoresorpatternsgivenbyhumanlanguageteachersPronunciationScoring/ErrorPatternDetection13Example:DialogueGameforPronunciationLearning14GoalofDialogueGame(1/3)CALL–CAPTNTUChineseoffersascoreandmulti-facetedcorrectivefeedbackstoeachpronunciationunit15learner/refcomparisonQualitativeassessmentCorrectivefeedback15GoalofDialogueGame(1/3)CALL–CAPTNTUChineseoffersascoreandmulti-facetedcorrectivefeedbackstoeachpronunciationunitDifferentlearnershaveverydifferentperformancedistributionsoverdifferentpronunciationunits16scorescorescorelearner1learner2learner3unitsunitsunits16WewishforeachindividuallearnerandeachpronunciationunitTheworsethescoreis,themorepracticeThehigherthescoreis,thelesscareNoneedforrepeatedpracticeonthesamesentence,butparticipatinginamoreinterestingdialoguegameTheneededpracticeopportunitiesautomaticallyappearsalongthedialogue17scorescorescoreunitsunitsunitsGoalofDialogueGame(2/3)17GoalofDialogueGame(3/3)Personalizedlearningmaterialsbasedonlearningstatusdynamicallyobtainedon-linealongthedialoguegameToachievethisgoalRecursivetree-structureddialoguescriptBestpathwithinthedialoguescriptforeachindividuallearnerfoundbyMDP1818DialogueGameScript(1/2)Tree-structuredturn-takingdialogueRestaurantscenario:seatingandmealordering19AasWaiter

BasCustomer19Tree-structuredturn-takingdialogueRestaurantscenario9sub-dialogueslinkedrecursivelyDialogueGameScript(2/2)20(8)Billpaying(1)Phoneinvitation(7)Chatting2(6)Mealserving(5)Chatting1(3)Meetinganddirections(4)Seatingandmealordering(2)Restaurantreservation(9)Sayinggoodbye..AlmostinfinitenumberofpathswithintherecursivetreesDifferentpathscontaindifferentdistributionsofthepronunciationunitsforpracticeDifferentpathsgoodfordifferentlearners20Basedontherecursivedialoguescript,thesystemprovidespersonalizedlearningmaterialsforeachindividuallearnerconsideringhislearningstatusSystemObjective(1/2)21Computer:User:Computer:User:

LearningStatus:ScoresofeachunitevaluatedbyNTUChinese92.0Scores85.8::66.349.1User:21Computer:User:Computer:User:SystemObjective(2/2)Thesystemselectson-linethepathwiththemostpracticeforthelower-scoredunitsforthelearnersofar,andreturnthecorrespondingnextsentencetopractice22Morepracticeinthepresentsentencedoesn’tnecessarilyimplythesameforthefuturesentencesalongthepathUser:22MarkovDecisionProcess(MDP)(1/4)StateslearningstatusofthelearnerRepresentedbyPresentdialogueturnLearner’saveragescoreforeverypronunciationunitsofar

(high-dimensionalcontinuousstatespace)23Unitbpmf…45Score53.489.074.780.3…97.0(1)(2)23MarkovDecisionProcess(MDP)(2/4)ActionsThesetofsentencestobeselectedforthelearnertopractice24User:Computer:User:Computer:User:24MarkovDecisionProcess(MDP)

(3/4)25Systemgoalsettingallpronunciationunits(orasubsetoffocusedunits)scored75orhigherover7timesforthelearnerinminimumnumberofdialogueturns,etc.Rewardsetcost-1foreverydialogueturn:thelessnumberofturnsthebetterGameendswhensystemgoalreachedPolicyBestsystemactiontotakeateachstatetobetrained(8)Billpaying(1)Phoneinvitation(7)Chatting2(6)Mealserving(5)Chatting1(3)Findingtherestaurant(4)Seatingandmealordering(2)Restaurantreservation(9)Sayinggoodbye25MarkovDecisionProcess(MDP)(4/4)

DialogueGame

MDPmodelS1S2STU:C:U:C:a1a2a3a42626MarkovDecisionProcess(MDP)(4/4)S1U:C:U:C:

DialogueGame

MDPmodel2727MarkovDecisionProcess(MDP)(4/4)S1U:C:U:C:DialogueGameMDPmodela1a2a3a42828MarkovDecisionProcess(MDP)(4/4)S1S2U:C:U:C:

DialogueGame

MDPmodela22929MarkovDecisionProcess(MDP)(4/4)S1S2U:C:U:C:RST

DialogueGame

MDPmodela2a13030LearnerSimulation(1/3)PolicytrainingneedssufficienttrainingdataSinceweneedreallearner’slanguagelearningbehaviorItisnoteasilyavailableLearnerSimulationModelisdevelopedforgeneratingalargenumberoftrainingdataReallearnerdata278learnersfrom36countries(balancedgender)Eachleanerrecorded30phoneticallybalancedsentences313132LearnerSimulationModel:GMMSimulatedLearne