微软亚洲主管研究员低资源场景下到语音合成

LowResource•••LowResource•••XuTan@MicrosoftResearch2SynthesizevoiceSynthesizevoicegiventext,letmachineto••••阅读教育:泛娱乐AI电话、AI主播、AITTSin•••AzureSpeechWindows,Office小冰(XiaoIce小娜(Cortanaetc,…DozensofTTSlanguages/localesXuTan@MicrosoftResearch3spiyspiyXuTan@MicrosoftResearch4spiyspiyprosodyfromtextSep.11th→SeptemberSpeech→spiy••••PolyphoneClassificationrising-fallingtone)模模模革命胜利后，又经受 一次次政治风浪的考验MoreaboutXuTan@MicrosoftResearch5spiyspiyprosodyfromtextGenerateacousticaccordingtolinguistic••••PolyphoneClassificationrising-fallingtone)••••TacotronDeepVoice1/2/3TransformerTTSMoreaboutNLPandXuTan@MicrosoftResearch6spiyspiyprosodyfromtextGenerateacousticaccordingtolinguisticSynthesizevoicegivenacoustic•••••Griffin-••••PolyphoneClassificationrising-fallingtone)••••TacotronDeepVoice1/2/3TransformerTTSMoreaboutNLPandSpeech(Signal7XuTan@MicrosoftResearchLowWhatisLowWhatisBigdataandbigLowresourcemainlyfocusesonlowdataandcomputation••XuTan@MicrosoftResearch8•••High•••HighqualityTextandaudio√TNrules,graphemetophonemedata,polyphone,prosodylabeled•OfflineOnlineservingand√XuTan@MicrosoftResearch9••Acousticmodel,fastonlineinference,forlowcomputation•LightTTS:TexttoSpeechwithLimitedPairedData(AlmostUnsupervisedTexttoSpeechandAutomaticSpeechRecognition),AcceptedtoICML2019Acousticmodel,languageexpansion,forlowdata•LightBERTforTTSfrontend,paperinXuTan@MicrosoftResearchxtEnd-to-endneuralTTSEnd-to-endneuralTTShasshownadvantagesoverconcatenativesynthesisandstatisticalparametricsynthesisEmphasis[1]/Tacotron[2]/DeepSlowinferencespeed:autoregressivemel-spectrogramgenerationisslowforlongNotrobust:wordsskippingandYoucancallmedirectlyat4257037344ormycell4254447474orsendmeameetingrequestwithalltheappropriateLietal,NeuralSpeechSynthesiswithTransformerNetwork.AAAIXuTan@MicrosoftResearchEnd-to-endneuralTTSEnd-to-endneuralTTShasshownadvantagesoverconcatenativesynthesisandstatisticalparametricsynthesisEmphasis[1]/Tacotron[2]/Deep•••Slowinferencespeed:autoregressivemel-spectrogramgenerationisslowforlongNotrobust:wordsskippingandLackofcontrollability:hardtocontrolthevoicespeedandprosodyintheautoregressiveLietal,NeuralSpeechSynthesiswithTransformerNetwork.AAAIXuTan@MicrosoftResearchTTSforonlineTTSforonlineTacotron/Transformermodelisslowforreal-timespeechsynthesis.(1svoicecosts1.2sonV100GPUusingTransformerTTS)Usingfast/expensive(e.g.,V100)foronlineservingisahugeCan••••servewithcheaperGPUorwithextremelyfasterwithrobustnessandcotrollability,withoutlossofvoicequality?XuTan@MicrosoftResearchKey•••Key•••Generatemel-spectrograminparallel(forRemovetheattentionmechanismbetweentextandspeech(forFeed-forwardtransformerwithlengthregulator(for•Extremelyfast:270xinferencespeeduponmel-spectrogramgeneration,speeduponvoiceRobust:nobadcaseofwordsskippingandControllable:cancontrolvoicespeedandVoicequality:onparorbetterthanSOTA•••XuTan@MicrosoftResearchPhoneme---->Mel-spectrogramPhoneme---->Mel-spectrogram----->Mel-••Feed-forwardtransformer:generatemel-spectrograminbothintrainingandinferenceRemovetheattentionmechanismbetweentextandspeechLengthRegulator:bridgethelengthmismatchbetweenphonemeandmelsequence(controllability)••XuTan@MicrosoftResearch•FFT(Feed-ForwardTransformer)•FFT(Feed-ForwardTransformer)•Replacedenseconnectionwithconvolutioninspeech•SharethesamemodelstructurethephonemesideandmelXuTan@MicrosoftResearch1.0xspiy223ssppiyiy1.0xspiy223ssppiyiyiy=0.5(111.50.5→1122232.0xspiyspiyiy=2(4462230.5xspiyssssppppiyiyiyiyiyiychXuTan@MicrosoftResearch223223•HowtogetthelabeltotraindurationExtractdurationbasedontheattentionalignmentsfromtheautoregressiveteacher223•XuTan@MicrosoftResearchXuTan@MicrosoftResearch270xspeedup270xspeedupformel-spectrogram38xspeedupforvoiceXuTan@MicrosoftResearchFastSpeechFastSpeechhasnorepeating,skippinganderrorYoucancallmedirectlyat4257037344ormycell4254447474orsendmeameetingrequestwithalltheappropriateinformation.Http0XX,Http1XX,Http2XX,XuTan@MicrosoftResearchFastSpeechFastSpeechachievescomparablevoicequalitywithTacotron2andTransformerTTS,andisclosetoground-truthrecordings.XuTan@MicrosoftResearchspiyspiy•Acousticmodel,fastonlineinference,forlowcomputation•LightTTS:TexttoSpeechwithLimitedPairedData(AlmostUnsupervisedTexttoSpeechandAutomaticSpeechRecognition),AcceptedtoICML2019Acousticmodel,languageexpansion,forlowdata•LightBERTforTTSfrontend,paperinXuTan@MicrosoftResearchLightTTS:LightTTS:TexttoSpeechwithLimitedPairedThereisstrongdemandtosupportmoreThereisstrongdemandtosupportmorelanguageswith•However,mostlanguageslackofpairedtextandspeechdata.Datalabelingalotofmoney(morethan$100000perour•HowtobuildTTSwithlimitedpairedLeverageunlabeledLeveragethedualnaturebetweenTTSandASRtohelpeachXuTan@MicrosoftResearchCommercializedTTSDenoisingauto-SpeechtotexttoBackDenoisingauto-SpeechtotexttoBackSpeechtotextBidirectionalsequence•Left-to- •Right-to-XuTan@MicrosoftResearchDenoisingauto-encoder(speechtospeech,textto•Weusedenoisingauto-encoder(DAE)Denoisingauto-encoder(speechtospeech,textto•Weusedenoisingauto-encoder(DAE)tolearnfromspeechandtext•N(x)meanstoaddnoiseon•Thebenefitsofdenoisingauto-Representationextraction:understandsequenceinspeechandtextXuTan@MicrosoftResearchDAEDAEIxxa IamaBacktransformation(texttospeech,speechBacktransformation(texttospeech,speechto•BacktransformationleveragesthedualnatureofTTSandASR,anddevelopcapabilityofspeech-textUseTTStoimproveUseASRtoimproveXuTan@MicrosoftResearchASRIlikeTTSIlikeTTSIamaASRIamaBidirectionalsequencemodeling(Left-to-right,right-to-•SequencegenerationBidirectionalsequencemodeling(Left-to-right,right-to-•Sequencegenerationsuffersfromerrorpropagation,especiallyforspeechwhichisusuallylongerthanThelaterpartofthesequenceisalwaysoflowerWeproposethebidirectionalsequencemodeling(BSM)thatgeneratesthesequenceinbothleft-to-rightandright-to-leftdirections.••XuTan@MicrosoftResearchASRIlikeASR.hceepsekilTTSIlikeTTS.hceepsekilupperlowerThehigher,theupperlowerThehigher,theThesmaller,the•••Ourmethodonlyleverages200pairedspeechandtextdata,andadditionalunpairedGreatlyoutperformsthemethodonlyusing200pairedClosetotheperformanceofsupervisedmethod(using12500pairedXuTan@MicrosoftResearchAudioDemo:XuTan@MicrosoftAudioDemo:XuTan@MicrosoftResearchPrintingthenforourpurposemaybeconsideredastheartofmakingbooksbymeansofmovabletypes.AfurtherdevelopmentoftheRomanlettertookplaceatVenice.spiyspiy•Acousticmodel,fastonlineinference,forlowcomputation•ICMLAcousticmodel,languageexpansion,forlowdata•LightBERTforTTSfrontend,paperinXuTan@MicrosoftResearchLightBERTLightBERTforTTS•Pre-trainingandfine-tuninghasbecomeanewparadigmforBERT/GPT-2/XLNet/RoBERTa/MASS•Pre-trainingandfine-tuninghasbecomeanewparadigmforBERT/GPT-2/XLNet/RoBERTa/MASSachievesgoodperformanceonmanylow-resourceBERTachievesgoodperformanceonTTSfrontendtasks,includingplyphoneclassification,prosodyprediction,etc.However,thepre-trainedmodelsaretoobigtobeservedonlineforpractical••e.g.,InferencetimeforBERTonpolyphoneclassificationOnlineenvironment:single-threadCPU,usuallylessthan20~30XuTan@MicrosoftResearchInfernecetimeBERT20250••Howtoincreaseinferencespeedunder••HowtoincreaseinferencespeedunderlimitedcomputationWeproposeatwo-stageknowledgedistillationmethodthattransferstheknowledgefrombigmodelintosmallmodelbothinpre-trainingandfine-tuningstage!BigPre-BigFine-knowledgeknowledgeSmallPre-SmallFine-XuTan@MicrosoftResearch••LightPAFF:ALightweightPre-trainingAndFine-tuningSupportmost••LightPAFF:ALightweightPre-trainingAndFine-tuningSupportmostpre-trainingandfine-tuningmodels,suchasBERT,GPT/GPT-2,XLNet,RoBERTa,MASS,XLM,MTDNN,UniLM,ERNIE,etc.Supportdifferentdownstreamtasks:languageunderstanding(BERT),languagemodeling/generation(GPT-2),seq2seqbasedlanguagegeneration(MASS)••BigPre-BigFine-knowledgeknowledgeSmallPre-SmallFine-XuTan@MicrosoftResearchResultsof•AccuracyandlatencyonTTSfrontendpolyphoneMeettheXuTan@MicrosoftResearchBERT-20250Resultsof•AccuracyandlatencyonTTSfrontendpolyphoneMeettheXuTan@MicrosoftResearchBERT-202503323withKDwithKDTransformer(pre-BERT-•••HighqualityTextand•••HighqualityTextandaudio？√TNrules,Graphemeandphonemedata,Polyphone,prosodylabeled？••Offline？√OnlineservingandXuTan@MicrosoftResearch•Light•••Smallermodel•Light•••Smallermodelsize,Leverageknowledgefromsignalprocessing,Generateinparallelinsteadofauto-regressively:ParallelWaveNet,WaveGlow,FloWaveNet,WaveFlow,GAN-TTS•LowdataresourceTTSin••••LabelingdataisChangecleandatatonoisydataChangesinglespeakertomulti-speakerCanwedonotusefrontendbutjustend-to-XuTan@MicrosoftResearch•••Interns:Yi•••Interns:YiRen,YangjunRuan,HaoCollaborators:ShengZhao,Jun-WeiGan,ZhouGuidances:TaoQin,Tie-Yan••AlltheonlineXuTan@MicrosoftResearchWeareMSRAWeareMSRAMachineLearningNMT,NLP,Spe