声学环境的鲁棒性

115.0Robustness

for

Acoustic

Environment声学环境的鲁棒性数字语音处理概论IntroductiontoDigitalSpeechProcessingReferences:1.10.5,10.6ofHuang 2.“RobustSpeechRecognitioninAdditiveandConvolutionalNoiseUsingParallel ModelCombination”ComputerSpeechandLanguage,Vol.9,1995 3.“AVectorTaylorSeriesApproachforEnvironmentIndependentSpeech Recognition”,InternationalConferenceonAcoustics,SpeechandSignal Processing,1996 4.“SignalBiasRemovalbyMaximumLikelihoodEstimationforRobustTelephone SpeechRecognition”,IEEETrans.onSpeech&AudioProcessing,Jan19965.“CepstralDomainSegmentalFeatureVectorNormalizationforNoiseRobust SpeechRecognition”,SpeechCommunication,Vol.25,pp.133-147,August19986.“Non-linearTransformationoftheFeatureSpaceforRobustSpeechRecognition”,inProceedingsofInternationalConferenceonAcoustics,Speech andSignalProcessing,2002,pp.401-404 7.“RASTAProcessingofSpeech”,IEEETrans.onSpeech&AudioProcessing, April1994 8.3.8ofDuda,HartandStork,“PatternClassification”,JohnWileyandsons,20019.“OptimizationofTemporalFiltersforConstructingRobustFeaturesinSpeech Recognition”,IEEETrans.onSpeechandAudioProcessing,May200610.“SuppressionofAcousticNoiseinSpeechUsingSpectralSubtraction”,IEEETrans. onAcoustics,SpeechandSignalProcessing,Apr197911.“APerceptuallyConstrainedGSVD-basedApproachforEnhancingSpeech CorruptedbyColorNoise”,IEEETransactionsonAudio,SpeechandLanguage Processing,Jan20072MismatchinStatisticalSpeechRecognitionMismatchbetweenTraining/RecognitionConditionsMismatchinAcousticEnvironmentEnvironmentalRobustnessadditive/convolutionalnoise,etc.MismatchinSpeakerCharacteristicsSpeakerAdaptationMismatchinOtherAcousticConditionsspeakingmode:read/prepared/conversational/spontaneousspeech,etc.speakingrate,dialects/accents,emotionaleffects,etc.MismatchinLexiconLexiconAdaptationout-of-vocabulary(OOV)words,pronunciationvariation,etc.MismatchinLanguageModelLanguageModelAdaptationdifferenttaskdomainsgivedifferentN-gramparameters,etc.PossibleApproachesforAcousticEnvironmentMismatch(training)(recognition)FeatureExtractionFeatureExtractionModelTrainingSearchandRecognitionAcousticModelsAcousticModelsO=o1o2…oTO’=o’1o’2…o’T

i=(Ai,Bi,

i)

’i=(A’i,B’i,

’i)SpeechEnhancementFeature-basedApproachesModel-basedApproachesy[n]x[n]h[n]acousticreceptionmicrophonedistortionphone/wirelesschanneln1(t)n2(t)FeatureExtractionSearchSpeechCorpusAcousticModelsLexiconLanguageModelTextCorpusy[n]O=o1o2…oTfeaturevectorsinputsignaladditivenoiseconvolutionalnoiseadditivenoiseoutputsentencesoriginalspeechx[n]W=w1w2...wR3Model-basedApproachExample1―ParallelModelCombination(PMC)BasicIdeaprimarilyhandlingtheadditivenoisethebestrecognitionaccuracycanbeachievedifthemodelsaretrainedwithmatchednoisyspeech,whichisimpossibleanoisemodelisgeneratedinreal-timefromthenoisecollectedintherecognitionenvironmentduringsilenceperiodcombiningthenoisemodelandtheclean-speechmodelsinreal-timetogeneratethenoisy-speechmodelsBasicApproachesperformedonmodelparametersincepstraldomainnoiseandsignalareadditiveinlinearspectraldomainratherthanthecepstraldomain,sotransformingtheparametersbacktolinearspectraldomainforcombinationallowingboththemeansand variancesofamodelsettobemodifiedParametersused:thecleanspeech

modelsanoisemodelModelcombinationNoiseHMMCleanspeechHMMLinearSpectraldomainNoisyspeechHMMCepstraldomainC-1expClog4processingprocessingParallelModelCombination(PMC)5Model-basedApproachExample1―ParallelModelCombination(PMC)TheEffectofAdditiveNoiseintheThreeDifferentDomainsandtheRelationshipsX(

)=S(

)+N(

)LinearpowerspectraldomainLogspectraldomainCepstraldomainNonlinearcombinationlogCC-1expXl=log(X),

Sl=log(S)

S=exp(Sl)

Nl=log(N)

N=exp(Nl)

Xc=CXl,

Sc=CSl

Sl=C-1Sc

Nc=CNl

Nl=C-1Nc

6Model-basedApproachExample1―ParallelModelCombination(PMC)TheStepsofParallelModelCombination(Log-NormalApproximation):basedonvariousassumptionsandapproximationstosimplifythemathematicsandreducethecomputationrequirementsCleanspeechHMM’sNoiseHMM’sCepstraldomainLog-spectraldomainLinearspectraldomainNoisyspeechHMM’s7Model-basedApproachExample2―VectorTaylor’sSeries(VTS)BasicApproachSimilartoPMC,thenoisy-speechmodelsaregeneratedbycombinationofcleanspeechHMM’sandthenoiseHMMUnlikePMC,thisapproachcombinesthemodelparametersdirectlyinthelog-spectraldomainusingTaylor’sSeriesapproximationTaylor’sSeriesExpansionforl-dimfunctions:

xcxf(x)f(x)

8Givenanonlinearfunctionz=g(x,y)x,y,zaren-dimrandomvectorsassumingthemeanofx,y,μx,μyandcovarianceofx,y,Σx,ΣyareknownthenthemeanandcovarianceofzcanbeapproximatedbytheVectorTaylor’sSeriesNowReplacingz=g(x,y)bytheFollowingFunctionthesolutioncanbeobtainedi,j:dimensionindexVectorTaylor’sSeries(VTS)9Feature-basedApproachExample1—CepstralMomentNormalization(CMS,CMVN)andHistogramEqualization(HEQ)CepstralMeanSubtraction(CMS)-OriginallyforCovolutionalNoiseconvolutionalnoiseintimedomainbecomesadditiveincepstraldomain(MFCC) y[n]=x[n]h[n]y=x+h, x,y,hincepstraldomainmostconvolutionalnoisechangesonlyveryslightlyforsomereasonabletimeinterval x=yh ifhcanbeestimatedCepstralMeanSubtraction(CMS)assumingE[

x

]=0, thenE[

y

]=h, averagedoveranutteranceoramovingwindow,oralongertimeinterval xCMS

=yE[

y

]CMSfeaturesareimmunetoconvolutionalnoise x[n]convolvedwithanyh[n]givesthesamexCMSCMSdoesn'tchangedeltaordelta-deltacepstralcoefficientsSignalBiasRemovalestimatinghbythemaximumlikelihoodcriteria h*=argmaxProb[Y=(y1y2…yT)|

,h],

:HMMfortheutteranceYiterativelyobtainedviaEMalgorithmCMS,CepstralMeanandVarianceNormalization(CMVN)andHistogramEqualization(HEQ)CMSequallyusefulforadditivenoiseCMVN:variancenormalizedaswellHEQ:thewholedistributionequalizedSuccessfulandpopularlyusedhxCMVN=xCMS/[Var(xCMS)]1/2y=CDFy-1[CDFx(x)]10CMSCMVNPx(x)Px(x)Px(x)Py(y)Py(y)Py(y)CepstralMomentNormalizationCMVN:variancenormalizedaswellxCMVN=xCMS/[Var(xCMS)]1/211probabilitydensityfunction(p.d.f.)CDFx(‧)CDFy(‧)xy1.0cumulativedistributionfunction(c.d.f.)HistogramEqualizationHEQ:thewholedistributionequalizedy=CDFy-1[CDFx(x)]12Feature-basedApproachExample2―RASTA(RelativeSpectral)TemporalFilteringTemporalFilteringeachcomponentinthefeaturevector(MFCCcoefficients)consideredasasignalor“timetrajectories”whenthetimeindex(framenumber)progressesthefrequencydomainofthissignaliscalledthe“modulationfrequency”performingfilteringonthesesignalsRASTAProcessing:assumingtherateofchangeofnonlinguisticcomponentsinspeech(e.g.additiveandconvolutionalnoise)oftenliesoutsidethetypicalrateofthechangeofthevocaltractshapedesigningfilterstotrytosuppressthespectralcomponentsinthese“timetrajectories”thatchangemoreslowlyorquicklythanthistypicalrateofchangeofthevocaltractshapeaspeciallydesignedtemporalfilterforsuch“timetrajectories”ModulationFrequency(Hz)MFCCFeaturesfeatureFrameindexB(z)B(z)B(z)vectorsytNewFeaturesFrameindex13x[n],x(t)t,nModulationFrequencyFrequencyC1[n]C3[n]C1[n]C3[n]FFTemporalFiltering

14Features-basedApproachExample3―Data-drivenTemporalFiltering(1)PCA-derivedtemporalfilteringtemporalfilteringisequivalenttotheweightedsumofasequenceofaspecificMFCCcoefficientwithlengthLslidedalongtheframeindexmaximizingthevarianceofsuchaweightedsumishelpfulinrecognitiontheimpulseresponseofBk(z)canbethefirsteigenvectorofthecovariancematrixforzk,forexampleBk(z)isdifferentfordifferentkFrameindexB1(z)B2(z)Bn(z)Lzk(1)zk(2)zk(3)Originalfeature

streamyt15filtering:convolution

Filtering16PCA(P.13of13.0)

17PCA(P.12of13.0)

18PrincipalComponentAnalysis(PCA)(P.11of13.0)ProblemDefinition:

forazeromeanrandomvectorxwithdimensionalityN,x∈RN,E(x)=0,iterativelyfindasetofk(k

N)orthonormalbasisvectors{e1,e2,…,ek}sothat

(1)var(e1Tx)=max(xhasmaximumvariancewhenprojectedone1)

(2)var(eiTx)=max,subjecttoei

ei-1……e1,2

i

k

(xhasnextmaximumvariancewhenprojectedone2,etc.)Solution:{e1,e2,…,ek}aretheeigenvectorsofthecovariancematrix

forxcorrespondingtothelargestkeigenvaluesnewrandomvectory

Rk:theprojectionofxontothesubspacespannedbyA=[e1

e2……

ek],y=ATxasubspacewithdimensionalityk≤Nsuchthatwhenprojectedontothissubspace,yis“closest”toxintermsofits“randomness”foragivenkvar(eiT

x)istheeigenvalueassociatedwithei

Proofvar(e1T

x)=e1TE(x

xT)e1=e1TΣe1=max,subjectto|e1|2=1usingLagrangemultiplier J(e1)=e1TE(xxT)e1-λ(|e1|2-1),

⇒E(xxT)e1=λ1e1,var(e1T

x)=λ1=maxsimilarfor

e2withanextraconstrainte2Te1=0,etc.=0

J(e1)

e119LinearDiscriminativeAnalysis(LDA)LinearDiscriminativeAnalysis(LDA)whilePCAtriestofindsome“principalcomponents”tomaximizethevarianceofthedata,theLinearDiscriminativeAnalysis(LDA)triestofindthemost“discriminative”dimensionsofthedataamongclasses/a//i/x2x1w120desiredundesiredLinearDiscriminativeAnalysis(LDA)within-classscattermatrix:

21LinearDiscriminativeAnalysis(LDA)tr(M):traceofamatrixM,thesumofeigenvalues,orthe“totalscattering”WTSB,WW:thematrixSB,Wafterprojectingonthenewdimensions

22LinearDiscriminativeAnalysis(LDA)Between-classscattermatrix:

23LinearDiscriminativeAnalysis(LDA)ProblemDefinitionwj,

jandUjaretheweight(ornumberofsamples),meanandcovariancefortherandomvectorsofclassj,j=1……N,μ

isthetotalmeanfindW=[w1w2……wk],asetoforthonormalbasissuchthattr(M):traceofamatrixM,thesumofeigenvalues,orthe“totalscattering” WTSB,WW:thematrixSB,WafterprojectingonthenewdimensionsSolutionthecolumnsofWaretheeigenvectorsofSw-1SBwiththelargesteigenvalues24Features-basedApproachExample3―Data-drivenTemporalFiltering(2)LDA/MCE-derivedTemporalFilteringFilteredparametersareweightedsumofparametersalongthetimetrajectory(orinnerproduct)Frameindex12345Foraspecifictime

trajectorykNewtimetrajectory

offeatureszk(1)zk(2)zk(3)(wk1,wk2,wk3)=wkTLDA/MCE-derivedfilterDividedintoclasses

3

2Class1zk

xk

=wkTzkLDA/MCEcriteria25SpeechEnhancementExample1―SpectralSubtraction(SS)SpeechEnhancementproducingabettersignalbytryingtoremovethenoiseforlisteningpurposesorrecognitionpurposesBackgroundNoisen[n]changesfastandunpredictablyintimedomain,butrelativelyslowlyinfrequencydomain,N(w)

y[n]=x[n]+n[n]SpectrumSubtraction|N(w)|estimatedbyaveragingoverMframesoflocallydetectedsilenceparts,orup-datedbythelatestdetectedsilenceframe

|N(w)|i=β|N(w)|i-1+(1-β)

|N(w)|i,n

|N(w)|i:|N(w)|usedatframei |N(w)|i,n:latestdetectedatframeisignalamplitudeestimation

|X(w)|i=|Y(w)|i-|N(w)|i,if|Y(w)|i-|N(w)|i>α|Y(w)|i

=α|Y(w)|i if|Y(w)|i-|N(w)|i≤α|Y(w)|i transformedbacktox[n]usingtheoriginalphaseperformedframebyframeusefulformostcases,butmayproducesome“musicalnoise”aswellmanydifferentimprovedversions^^26N(ω)ω

ω1ω2

ω3Y(ω)SpectralSubtraction27SpeechEnhancementExample2―SignalSubspaceApproachSignalSubspaceApproachrepresentingsignalplusnoiseasavectorinaK-dimensionalspacesignalsareprimarilyspannedinam-dimensionalsignalsubspacetheotherK-mdimensionsareprimarilynoiseprojectingthereceivednoisysignalontothesignalsubspacesignalplusnoiseprojectedonsignalsubspacesignalplusnoise(K-dim)cleanspeechsignalsubspace(m-dim)28SpeechEnhancementExample2―SignalSubspaceApproachAnExampleHankel-formmatrix

signalsamples:y1y2y3…yk…yL…yM

y1

y2y3y4…yk

y2

y3y4……yk+1

Hy=y3

y4……….yk+2

.