语音识别更新90

9.0SpeechRecognitionUpdates语音识别更新数字语音处理概论IntroductiontoDigitalSpeechProcessing1Minimum-Classification-Error(MCE)andDiscriminativeTrainingAPrimaryProblemwiththeConventionalTrainingCriterion:Confusingsets

find

(i)suchthatP(X|

(i))ismaximum(MaximumLikelihood)ifX

CiThisdoesnotalwaysleadtominimumclassificationerror,sinceitdoesn'tconsiderthemutualrelationshipamongcompetingclassesThecompetingclassesmaygivehigherlikelihoodfunctionforthetestdataGeneralObjective:findanoptimalsetofparameters(e.g.forrecognitionmodels)tominimizetheexpectederrorofclassificationthestatisticsoftestdatamaybequitedifferentfromthatofthetrainingdata

trainingdataisneverenoughAssumetherecognizerisoperatedwiththefollowingclassificationprinciples:

{Ci,i=1,2,...M},Mclasses

(i):statisticalmodelforCi ={(i)}i=1……M,thesetofallmodelsforallclasses X:observations

gi(X,

):classconditionedlikelihoodfunction,forexample,

gi(X,

)=P(X|

(i))C(X)=Ci

ifgi(X,

)=maxjgj(X,

):classificationprinciplesanerrorhappenswhen

P(X|

(i))=maxbutX

Ci

2

(0),(1),(2),…(9)P(O|

(k))

P(O|

(7))：correctP(O|

(1))：competingwrongMinimum-Classification-Error(MCE)3Minimum-Classification-Error(MCE)TrainingOneformofthemisclassificationmeasure

Comparisonbetweenthelikelihoodfunctionsforthecorrectclassandthecompetingclasses

Acontinuouslossfunctionisdefinedl(d)→0whend→-∞ l(d)→1whend→∞ θ:switchingfrom0to1nearθ

γ:determiningtheslopeatswitchingpointOverallClassificationPerformanceMeasure:4

0γ1γ2SigmoidFunction

5Minimum-Classification-Error(MCE)TrainingFind

suchthattheaboveobjectivefunctioningeneralisdifficulttominimizedirectlylocalminimumcanbeobtainediterativelyusinggradient(steepest)descentalgorithmeverytrainingobservationmaychangetheparametersofALLmodels,notthemodelforitsclassonly^

partialdifferentiationwithrespecttoalldifferentparametersindividuallyt:thet-thiterationε:adjustmentstepsize,shouldbecarefullychosen6a1a2L(a1)GradientDescentAlgorithm7DiscriminativeTrainingandMinimumPhoneErrorRate(MPE)TrainingForLargeVocabularySpeechRecognitionMinimumBayesianRisk(MBR) adjustingallmodelparameterstominimizetheBayesianRiskΛ:{λi,i=1,2,……N}acousticmodelsΓ:LanguagemodelparametersOr:r-thtrainingutterancesr:correcttranscriptionofOrBayesianRisku:apossiblerecognitionoutputfoundinthelatticeL(u,sr):LossfunctionPΛ,Γ

(u|Or):posterioriprobabilityofugivenOrbasedonΛ,Γ

OtherdefinitionsofL(u,sr)possibleMinimumPhoneErrorRate(MPE)Training

Acc(u,sr):phoneaccuracyBetterfeaturesobtainableinthesamewaye.g.yt=xt+Mht feature-spaceMPE8MinimumPhoneError(MPE)RateTrainingLatticePhoneAccuracyTimeReferencephonesequenceDecodedphonesequenceforapathinthelattice9ReferencesforMCE,MPEandDiscriminativeTraining“MinimumClassificationErrorRateMethodsforSpeechRecognition”,IEEETrans.SpeechandAudioProcessing,May1997“SegmentalMinimumBayes-RickDecodingforAutomaticSpeechRecognition”,IEEETrans.SpeechandAudioProcessing,2004“MinimumPhoneErrorandI-smoothingforImprovedDiscriminativeTraining”,InternationalConferenceonAcoustics,SpeechandSignalProcessing,2002“DiscriminativeTrainingforAutomaticSpeechRecognition”,IEEESignalProcessingMagazine,Nov201210SubspaceGaussianMixtureModel

Gaussian1Gaussian2GaussianI

…Gaussian1Gaussian2GaussianI

…Gaussian1Gaussian2GaussianI

……(HMMState)jsubstateweightvector11SubspaceGaussianMixtureModelAtriphoneHMMinSubspaceGMM…v1…v2…HMMState…v3…v4…HMMState…v5…v6…HMMStateSubstateGaussianHMM

…

SharedParametersShared

12SubspaceGaussianMixtureModelAtriphoneHMMinSubspaceGMM…v1…v2…HMMState…v3…v4…HMMState…v5…v6…HMMStateSubstateGaussianHMM

…

SharedParameters………=

Miisthebasissetspanningasubspaceofmean(columnsofMi

notnecessarilyorthogonal)

13SubspaceGaussianMixtureModelAtriphoneHMMinSubspaceGMM…v1…v2…HMMState…v3…v4…HMMState…v5…v6…HMMStateSubstateGaussianHMM

…

SharedParametersThelikelihoodofHMMstatejgivenot

j:state,m:substate,i:Gaussian14ReferencesforSubspaceGaussianMixtureModel"TheSubspaceGaussianMixtureModel–aStructuredModelforSpeechRecognition",

D.Povey,LukasBurgetet.alComputerSpeechandLanguage,2011"ASymmetrizationoftheSubspaceGaussianMixtureModel",

DanielPovey,MartinKarafiat,ArnabGhoshal,PetrSchwarz,ICASSP2011"SubspaceGaussianMixtureModelsforSpeechRecognition",

D.Povey,LukasBurgetetal.,ICASSP2010

"ATutorial-StyleIntroductionToSubspaceGaussianMixtureModelsForSpeechRecognition",

MicrosoftResearchtechnicalreport

MSR-TR-2009-11115NeuralNetwork—ClassificationTaskClassifierMaleFemaleOthersClassifierFeaturesClassesHairLengthMake-up...16HairLengthMake-UpFemaleMaleVoicepitchNeuralNetwork—2DFeatureSpace17NeuralNetwork‒Multi-DimensionalFeatureSpaceWeneedsometypeofnon-linearfunction!

18NeuralNetwork—NeuronsEachneuronreceivesinputsfromotherneuronsTheeffectofeachinputontheneuronisadjustable(weighted)Theweightsadaptsothatthewholenetworklearnstoperformusefultasks19w4yx4x3x2x1w3w2w11bNeuralNetworkAlotofsimplenon-linearitycomplexnon-linearity20NeuralNetworkTraining–BackPropagationStartwithrandomweightsComparetheoutputsofthenettothetargetsTrytoadjusttheweightstominimizetheerror14-30.20.901tjTargetwyj21GradientDescentAlgorithm

22LearningrateUpdatedweightsErrorw4x4x3x2x1w3w2w1GradientDescentAlgorithmWeightatt-thiterationw1w223NeuralNetwork—FormalFormulation

24ReferencesforNeuralNetworkRumelhart,DavidE.;Hinton,GeoffreyE.,Williams,RonaldJ."Learningrepresentationsbyback-propagatingerrors".Nature,1986.Alpaydın,Ethem.Introductiontomachinelearning(2nded.),MITPress,2010.AlbertNigrin,NeuralNetworksforPatternRecognition(1sted.).ABradfordBook,1993.Reference:

Neural

Networks

for

Machine

Learning

course

by

Geoffrey

Hinton,

Coursera25Spectrogram26Spectrogram27GaborFeatures(1/2)

28GaborFeatures(2/2)29IntegratingHMMwithNeuralNetworksTandemSystemMulti-layerPerceptron(MLP,orNeuralNetwork)offersphonemeposteriorvectors(posteriorprobabilityforeachphoneme)MLPtrainedwithknownphonemesforMFCC(orplusGabor)vectorsforoneorseveralconsecutiveframesastargetphonemeposteriorsconcatenatedwithMFCCasanewsetoffeaturesforHMMphonemeposteriorprobabilitiesmayneedfurtherprocessingtobebettermodeledbyGaussiansHybridSystemGaussianprobabilitiesineachtriphoneHMMstatereplacedbystateposteriorsforphonemesfromMLPtrainedbyfeaturevectorswithknownstatesegmentation30PhonemePosteriorsandStatePosteriorsNeuralNetworkTraining

PhonePosteriorStatePosterior

31IntegratingHMMwithNeuralNetworksTandemSystemphonemeposteriorvectorsfromMLPconcatenatedwithMFCCasanewsetoffeaturesforHMMFeatureExtractionMLPconcatenationHMMTrainingAcousticModelsLexiconLanguageModelDecodingandsearchInputspeechoutputMFCCGabor32IntegratingHMMwithNeuralNetworksTandemSystemphonemeposteriorvectorsfromMLPconcatenatedwithMFCCasanewsetoffeaturesforHMMFeatureExtractionMLPconcatenationHMMTrainingAcousticModelsLexiconLanguageModelDecodingandsearchInputspeechoutputMFCCGabor33ReferencesReferencesforGaborFeaturesandTandemSystemRichardM.Stern&NelsonMorgan,“HearingIsBelieving”,IEEESIGNALPROCESSINGMAGAZINE,NOVEMBER2012Hermansky,H.,Ellis,D.P.W.,Sharma,S.,“TandemConnectionistFeatureExtractionForConventionalHmmSystems”,inProc.ICASSP2000.Ellis,D.P.W.andSingh,R.andSivadas,S.,“Tandemacousticmodelinginlarge-vocabularyrecognition”,inProc.ICASSP2001.“ImprovedTonalLanguageSpeechRecognitionbyIntegratingSpectro-TemporalEvidenceandPitchInformationwithProperlyChosenTonalAcousticUnits”,Interspeech,Florence,Italy,Aug2011,pp.2293-2296.34DeepNeuralNetwork(DNN)

35RestrictedBoltzmannMachineRestrictedBoltzmannMachine(RBM):agenerativemodelforprobabilityofvisibleexamples(p(v))withahiddenlayerofrandomvariables(h)topology:undirectedbipartitegraphW:weightmatrix,describingcorrelationbetweenvisibleandhiddenlayersa,b:biasvectorsforvisibleandhiddenlayersE:energyfunctionfora(v,h)pairRBMtraining:adjustingW,a,andbtomaximizep(v)Property:findingagoodrepresentation(h)forvinunsupervisedmannerUsinglargequantitiesofunlabelleddata36RBMInitializationforDNNTrainingRBMInitializationweightmatricesofDNNinitializedbyweightmatrixesofRBMsaftertraininganRBM,generatesamplesinhiddenlayerusedfornextlayerofRBMstepsofinitialization(e.g.3hiddenlayers)1.RBMtraining2.sampling……3.RBMtraining…4.sampling5.RBMtraining6.copyweightandbiasasinitializationinputsamplesDNN7.backpropagation37DeepNeuralNetworkforAcousticModelingDNNastriphonestateclassifierinput:acousticfeatures,e.g.MFCCoutputlayerofDNNrepresentingtriphonestatesfinetuningtheDNNbybackpropagationusinglabelleddataHybridSystemnormalizedoutputofDNNasposteriorofstatesp(s|x)statetransitionremainingunchanged,modeledbytransitionprobabilitiesofHMM…s1s2sn…a11a12a22annMFCCframes(x)DNNHMM38BottleneckFeaturesfromDNN…………xiSizeofoutputlayer

=No.ofstates…………

P(a|xi)P(b|xi)P(c|xi)AcousticfeatureDNN39ReferencesforDNNContext-DependentPre-trainedDeepNeuralNetworksforLargeVocabularySpeechRecognitionGeorgeE.Dahl,DongYu,DengLi,andAlexAceroIEEE

Trans.onAudio,SpeechandLanguageProcessing,Jan,2012AfastlearningalgorithmfordeepbeliefHinton,G.E.,Osindero,S.andTeh,YNeuralComputation,

18,pp1527-1554,

2006DeepNeuralNetworksforAcousticModelinginSpeechRecognitionG.Hinton,L.Deng,D.Yu,G.Dahl,A.Mohamed,N.Jaitly,A.Senior,V.Vanhoucke,P.Nguyen,T.Sainath,andB.KingsburyIEEESignalProcessingMagazine,

29,November2012DeepLearningandItsApplicationstoSignalandInformationProcessingIEEESignalProcessingMagazine,Jan2011ImprovedBottleneckFeaturesUsingPretrainedDeepNeuralNetworksYu,Dong,andMichaelL.SeltzerInterspeech2011Extractingdeepbottleneckfeaturesusingstackedauto-encodersGehring,Jonas,etal.ICASSP201340ConvolutionalNeuralNetwork(CNN)SuccessfulinprocessingimagesSpeechcanbetreatedasimagesSpectrogramTimeFrequency41a2a1b1b2MaxMaxMaxpoolingConvolutionalNeuralNetwork(CNN)Anexample42CNNConvolutionalNeuralNetwork(CNN)Anexample43CNNImageCNNReplaceDNNbyCNNConvolutionalNeuralNetwork(CNN)ProbabilitiesofstatesAnexample44MemoryCellLongShort-termMemory(LSTM)InputGateOutputGateSignalcontroltheinputgateSignalcontroltheoutputgateForgetGateSignalcontroltheforgetgateOtherpartofthenetworkOtherpartofthenetwork(Otherpartofthenetwork)(Otherpartofthenetwork)(Otherpartofthenetwork)LSTMSpecialNeuron:4inputs,1output45

multiplymultiply

between0and1foropeningandclosingthegatec

LongShort-termMemory(LSTM)46x1x2InputSimplyreplacingtheneuronswithLSTMoriginalnetwork

…………

LongShort-termMemory(LSTM)47x1x2++++++++Input

4timesofparametersLongShort-termMemory(LSTM)48ReferencesLongShort-termMemory(LSTM)Graves,N.Jaitly,A.Mohamed.“HybridSpeechRecognitionwithDeepBidirectionalLSTM”,ASRU2013.Graves,Alex,andNavdeepJaitly."Towardsend-to-endspeechrecognitionwithrecurrentneuralnetworks."

Proceedingsofthe31stInternationalConferenceonMachineLearning(ICML-14).2014.ConvolutionalNeuralNetwork(CNN)ConvolutionalNeuralNetworkforImageprocessingZeiler,M.D.,&Fergus,R.(2014).“Visualizingandunderstandingconvolutionalnetworks.”In

ComputerVision–ECCV2014

ConvolutionalNeuralNetworkforspeechprocessingTóth,László."Convolutionaldeepmaxoutnetworksforphonerecognition."Proc.Interspeech.2014.ConvolutionalNeuralNetworkfortextprocessingShen,Yelong,etal."Alatentsemanticmodelwithconvolutional-poolingstructureforinformationretrieval."

Proceedingsofthe23rdACMInternationalConferenceonConferenceonInformationandKnowledgeManagement.ACM,2014.49

NeuralNetworkLanguageModelingvocabularysize50x(t):inputlayery(t):outputlayers(t):hiddenlayerPreviousword,using1-of-Nencoding000………001000…

Vocab.sizeRecursivestructurepreserveslong-termhistoricalcontext.Probabilitydistributionofnextword,vocabularysize.RecurrentNeuralNetworkLanguageModeling(RNNLM)VU51RNNLMStructurex

52BackpropagationforRNNLMUnfoldrecurrentstructureInputonewordatatimeDonormalbackpropagationunfoldthroughtimeYoshuaBengio,RejeanDucharmeandPascalVincent.“Aneuralprobabilisticlanguagemodel,”JournalofMachineLearningResearch,3:1137–1155,2003HolgerSchwenk.“Continuousspacelanguagemodels,”ComputerSpeechandLanguage,

vol.21,pp.492–518,2007TomášMikolov,MartinKarafiát,LukášBurget,JanČernockýandSanjeevKhudanpur.“Recurrentneuralnetworkbasedlanguagemodel,”inInterspeech2010MikolovTomášetal,“ExtensionsofRecurrentNeuralNetworkLanguageModel”,ICASSP2011.MikolovTomášetal,“ContextDependentRecurrentNeuralNetworkLanguageModel”,IEEESLT2012.ReferencesforRNNLM54WordVectorRepresentations(WordEmbedding)z1z2dogcatrabbit…1-of-Nencodingofthewordwi-1

……100Theprobabilityforeachwordasthenextwordwi……z1z2UsetheinputoftheneuronsinthefirstlayertorepresentawordwjumprunflowertreeWordvector,wordembeddingfeature:V(w)Wordanalogytask:(king)-(man)+(woman)→(queen)……55……____wi

____……WordVectorRepresentations–VariousArchitecturesContinuousbagofword

(CBOW)modelSkip-gram……wi-1

____wi+1……NeuralNetworkwiwi-1wi+1NeuralNetworkwi-1wiwi+1predictingthewordgivenitscontextpredictingthecontextgivenaword56ReferencesforWordVectorRepresentationsTomasMikolov,KaiChen,GregCorrado,andJeffreyDean.

”EfficientEstimationofWordRepresentationsinVectorSpace.”InProceedingsofWorkshopatICLR,2013.TomasMikolov,IlyaSutskever,KaiChen,GregCorrado,andJeffreyDean.

”DistributedRepresentationsofWordsandPhrasesandtheirCompositionality.”InProceedingsofNIPS,2013.TomasMikolov,Wen-tauYih,andGeoffreyZweig.

”LinguisticRegularitiesinContinuousSpaceWordRepresentations.”InProceedingsofNAACLHLT,2013.57WeightedFiniteStateTransducer(WFST)FiniteStateMachineAmathematicalmodelwiththeoriesandalgorithmsusedtodesigncomputerprogramsanddigitallogiccircuits,whichisalsocalled“FiniteAutomaton”.Thecommonautomataareusedasacceptors,whichcanrecognizeitslegalinputstrings.AcceptorAcceptanylegalstring,orrejectitEX:{ab,aab,aaab,...}=aa*bTransducerAfinitestatetransducer(FST)isaextensiontoFSATransduceanylegalinputstringtoanotheroutputstring,orrejectitEX:{aaa,aab,aba,abb}->{bbb,bba,bab,baa}WeightedFiniteStateMachineFSMwithweightedtransitionAnexampleofWFSATwopathsfor“ab”Throughstates(0,1,1);costis(0+1+2)=3Throughstates(0,2,4);costis(1+2+2)=5finalstateinitialstateoutputinputweight58WFSTOperations(1/2)

59WFSTOperations(2/2)MinimizationTheequivalentautomatonwithleastnumberofstatesandleasttransitionsWeightpushingRe-distributingweightamongtransitionswhilekeptequivalenttoimprovesearch(futuredevelopmentsknownearlier,etc.),especiallyprunedsearchWeightPushingMinimization60WFSTforASR(1/6)HCLG≡H◦C◦L◦GistherecognitiongraphGisthegrammarorLM(anacceptor)ListhelexiconCaddsphoneticcontext-dependencyHspecifiestheHMMstructureofcontext-dependentphonesInputOutputHCLGHMMstatesequencetriphonePhonemesequencewordtriphonephonemewordword61WFSTforASR(2/6)TransducerH:HMMtopologyInput:HMMstatesequenceOutput:context-dependentphoneme(e.g.,triphone)Weight:HMMtransitionprobability/a00

62WFSTforASR(3/6)TransducerC:context-dependencyInput:context-dependentphoneme(triphone)Output:context-independentphoneme(phoneme)$aba

ab63WFSTforASR(4/6)TransducerL:lexiconInput:context-independentphoneme(phoneme)sequenceOutput:wordWeight:pronunciationprobability

64WFSTforASR(5/6)AcceptorG:N-grammodelsBigramEachwordhasastateEachbigramw1w2hasatransitionw1tow2Introducingback-offstatebforback-offestimation.Anunseenw1w3bigramisrepresentedastwotransitions:anε-transitionfromw1tobandatransitionfrombtow3.65WFSTforASR(6/6)

frame2frame3frame166ReferencesWFSTMehryarMohri,“Finite-statetransducersinlanguageandspeechprocessing,”Comput.Linguist.,vol.23,no.2,pp.269–311,1997.WFSTforLVCSRMehryarMohri,FernandoPereira,andMichaelRiley,“Weightedautomataintextandspeechprocessing,”inEuropeanConferenceonArtificialIntelligence.1996,pp.46–50,JohnWileyandSons.MehryarMohri,FernandoC.Pereira,andMichaelRiley,“SpeechRecognitionwithWeightedFinite-StateTransducers,”inSpringerHandbookofSpeechProcessing,JacobBenesty,MohanM.Sondhi,andYitengA.Huang,Eds.,pp.559–584.SpringerBerlinHeidelberg,Secaucus,NJ,USA,2008.67ProsodicFeatures(І)P1P2d1d2Pitch-relatedFeatures(examplesinMandarinChinese)Theaveragepitchvaluewithinthesyllable

Themaximumdifferenceofpitchvaluewithinthesyllable

TheaverageofabsolutevaluesofpitchvariationswithinthesyllableThemagnitudeofpitchresetforboundaries

Thedifferenceofsuchfeaturevaluesofadjacentsyllableboundaries(P1-P2,d1-d2,etc.)atleast50pitch-relatedfeatures68Duration-relatedFeatures(examplesinMandarinChinese)atleast40duration-relatedfeaturesEnergy-relatedFeaturessimilarlyobtainedsyllableboundarysyllableboundarypausepauseendofutterancebeginofutteranceABCDEbaProsodicFeatures(Ⅱ)PausedurationbAveragesyllableduration

(B+C+D+E)/4or((D+E)/2+C)/2Averagesyllabledurationratio

(D+E)/(B+C)or

(D+E)/2/CCombinationofpause&syllable

features(ratioorproduct)

C*b,D*b,C/b,D/bLengthening

C/((A+B)/2)Standarddeviationoffeaturevalues69RandomForestforToneRecognitionforMandarinRandomForestalargenumberofdecisiontreeseachtrainedwitharandomlyselectedsubsetoftrainingdataand/orarandomlyselectedsubsetoffeaturesdecisionfortestdatabyvotingofalltrees•••70RecognitionFrameworkwithProsodicModelingRescoringFormula:λl,λp:weightingcoefficientsProsodicmodelAnexampleapproach:Two-passRecognition71ReferencesProsody“ImprovedLargeVocabularyMandarinSpeechRecognitionbySelectivelyUsingToneInformationwithaTwo-stageProsodicModel”,Interspeech,Brisbane,Australia,Sep2008,pp.1137-1140“LatentProsodicModeling(LPM)forSpeechwithApplicationsinRecognizingSpontaneousMandarinSpeechwithDisfluencies”,InternationalConferenceonSpokenLanguageProcessing,Pittsburgh,U.S.A.,Sep2006.“ImprovedFeaturesandModelsforDetectingEditDisfluenciesinTranscribingSpontaneousMandarinSpeech”,IEEETransactionsonAudio,SpeechandLanguageProcessing,Vol.17,No.7,Sep2009,pp.1263-1278.72PersonalizedRecognizerandSocialNetworksPersonalizedrecognizerisfeasibletodaySmartphoneuserispersonal

eachsmartphoneusedbyasingleuseruseridentificationisknownoncethesmartphoneisturnedonPersonalcorpusisavailableAudiodataeasilycollectedatserverTextdataavailableonsocialnetworks73RecognitionModuleintheCloud

Personal-izedLMPersonal-izedAMRecognitionEngineAcousticModelAdaptationPersonalizedAcousticDataLanguageModelAdaptationSocialNetworkCorporaWebCrawlerClientSpeechFriend1Friend2Friend3SocialNetworkCloudPosttranscriptionsUser’sWallTranscriptionsPersonalizedRecognizerandSocialNetworks74LanguageModelAdaptationFrameworkSocialNetworkCloudH:PersonalCorporaCollectionuser3user2user1user4user5user6targetuTrainingIntermediateLM(s)MaximumLikelihoodInterp.Consolida-tionBackgroundLMPersonalizedLMPersonalizedAMtargetuDevelopRecognitionEngine75ReferencesforPersonalizedRecognizer“PersonalizedLanguageModelingbyCrowdSourcingwithSocialNetworkDataforVoiceAccessofCloudApplications”,IEEESLT2012.“RecurrentNeuralNetworkBasedLanguageModelPersonalizationbySocialNetworkCrowdsourcing”,Interspeech2013.76RecognizingCode-switchedSpeechDefinitionCode-switchingoccursfromwordtowordinanutteranceExample:当我们要作

FourierTransform

的时候“Host”language“Guest”languageCode-switchedSystemViterbiDecoding这个complexity很高我买了iPad的配件SpeechUtteranceMandarinEnglishAcousticModelMandarinEnglishLanguageModelMandarinEnglishLexiconSpeechRecognitionBilingualacousticmodels,languagemodel,andlexiconAsignalframemaybelongtoaMandarinphonemeoranEnglishphoneme,aMandarinphonememaybeprecededorfollowedbyanEnglishphonemeandviceversa,aChinesewordmaybeprecededorfollowedbyanEnglishwordandviceversa(bilingualtriphones,bilingualn-grams,etc.)77RecognizingCode-switchedSpeechCode-switchingissuesImbalanceddatadistributionTherearemuchmoredataforhostlanguagebutonlyverylimitedforguestlanguageThemodelsforguestlanguageareusuallyweak,thereforeaccuracyislowInter-lingualambiguitySomephonemesfordifferentlanguagesareverysimilarbutdifferent(e.g.

ㄅvs.B),butmaybeproducedverycloselybythesamespeakerLanguageidentification(LID)UnitsforLIDaresmallerthananutteranceVerylimitedinformationisavailable这里是在讲FourierTransform的性质MandarinMandarinEnglishLanguageIdentification78RecognizingCode-switchedSpeechSomeapproachestohandletheaboveproblemsAcousticunitmergingandrecoverySomeacousticunitssharedacrosslanguages:Gaussian,state,modelSharedtrainingdataModelsrecoveredwithrespectivedatatopreservethelanguageidentityFrame-levellanguageidentification(LID)LIDforeachframeIntegratedinrecognitionGaussian1GaussianMStateTriphoneTriphoneTriphoneTriphoneTriphoneTriphoneGaussian1GaussianStateGaussianMStateGaussian1GaussianMStateViterbiDecodingProcedureBilingualLanguageModelBilingualLexiconBilingualAcousticModelLanguageDetectorCode-mixedSpeechBilingualTranscriptionIntegrationofLanguageIdentificationandSpeechRecognition79ReferencesforRecognizingCode-switchedSpeech“BilingualAcousticModelAdaptationByUnitMergingOnDifferentLevelsAndCross-levelIntegration,”

Interspeech,2011.“RecognitionOfHighlyImbalancedCode-mixedBilin-gualSpeechWithFrame-levelLanguageDetectionBasedOnBlurredPosteriorgram,”

ICASSP,2012.“LanguageIndependentAndLanguageAdaptiveAcousticModeling