create photorealistic talking face

1、Create Photo-Realistic Talking FaceChangbo Hu*This work was done during visiting Microsoft Research China with Baining Guo and Bo ZhangOutlineIntroduction of talking faceMotivationsSystem overviewTechniquesConclusionsIntroductionWhat is a talking faceFace (lip) animation, driven by voiceApplications

2、The process of talking faceFace modelMotion captureMapping between audio and video Rendering, Photo-realistic?LiteraturesWalter,93, DecFace, 2Dwire frame modelTerzopoulos,95, Skin and muscle modelBreglar,97, Video Rewrite, Sample image based TS Huang,98,Mesh model from range dataPoggio,98, MikeTalk,

3、 Viseme morphingGuenter,99, Making face, 3D from multicamera Zhengyou Zhang, 00, 3D face modeling from video through epipolar constraintCosatto,00, Planar quads model Some Face modelsMotivationsAim: a graphics interface for conversation agentPhoto-realisticDriven by ChineseSmooth connection between

4、sentencesExtended from “Video rewrite”System overview:Pipeline of the system(1)Video with SoundImagesSoundPose trackingPhoneme segmentationAnnotationLip motion TrackingTrain databaseSystem overview: Pipeline of the system(2)New textWav soundTTS systemTriphone sequenceSegmentationSynthesized triphone

5、 sequenceTrain databaseLip motion sequenceRewrite to facesBackground sequenceTechniquesAnalysis:Audio processImage processSynthesisLip image Background imageStitch togetherAudio part:Sound SegmentationGiven the wav file and the scriptUsing HMM to train the segment systemSegment wav file to phoneme s

6、equenceExample of the segmentation result:SILOPEN023SILOPEN2442s4361if46274j7580ia18197sh98109ang1110121y122130e4131133y134145in2146154h155164ang2165194Annotation with PhonemeUsing phoneme to annotate video framesEach phoneme in a sentence corresponds to a short time of video sequenceTraining Senten

7、ceAudio FramesVideo FramesPhoneme SequenceFrames for Phoneme1Frames for Phoneme1Phoneme1Frames for Phoneme2Frames for Phoneme2Phoneme2Phoneme Distance Analysis Phoneme&triphone basicsChinese Phoneme vs. English PhonemeDistance Metrics definitionsResultsPhoneme BasicsPhonemes represents the basic ele

8、ments in speech. All possible speech can be represented by combination of phonemes.CH, JH, S, EH, EY, OY, AE, SILTriphone are three consecutive phonemes. It not only represents pronounce characteristics but also contains context information.T-IY-P, IY-P-AA, P-AA-TChinese Phoneme vs. EnglishChinese p

9、honeme has two basic groups: Initials and Finals.Initials: B, P, M, F, Finals: a3, o1, e2, eng3, iang4, ue5, Chinese finals each has 5 tones: 1,2,3,4,5.Different tones: a1, a2, a3, a4, a5.Chinese finals actually is not a basic elements of speech.For example: iang1, iao1, uang1, iong1Chinese phoneme

10、set is much larger than English.Phoneme Distance AnalysisDefine the distance between any two phonemes.Since we only synthesis video but not sound, so tone is ignoredLip shape motion is the core element for distance metrics.Phoneme Distance AnalysisVideo 1Video 2Video 4Video 1Video 2Video 3Phoneme 1:

11、Phoneme 2:Time Align to an uniform lengthVideo 2Video 3Video 4Video 2Video 1Video 1Average the videos to get an average videoVideo AverageVideo AverageBy comparing the two aligned average videos, we generate the distance matrix of the whole phoneme set.Image part: Pose TrackingAssume a plane model f

12、or faceStandard minimization method to find transform matrix (affine transform)Black,95Mask is used to constrain interests part of the faceTemplate PictureMask ImagePose trackingMotion prediction using parameters with physical meaningPose TrackingSome tracking results:Lip Motion TrackingUsing Eigen

13、Points (Covell, 91)Feature Points include Jaw, lip and teethTraining database specified manuallyAuto tracking through all pose-tracked imagesLip motion trackingLip Motion TrackingTrain Database (hand-labeled)Auto Tracking ResultsSynthesis new sentencesNew text converted by TTS system to wavWav is se

14、gmented to phoneme sequenceUsing DP to find an optimal video sequence from the training databaseTime-align triphone videos and stitch them together.Transform the lip sequence and paste them to background faces.Lip sequence synthesisOptimal phoneme sequencesTriphone 1Triphone 2Triphone 5Triphone 3Tri

15、phone 4Triphone 6Triphone 7Triphone 8Triphone BTriphone 9Triphone ATriphone CNew phoneme sequencesNew phoneme sequencesDynamic ProgrammingBeginTriphone1Triphone3Triphone2Triphone4EndTriphone5Edge Cost DefinitionTwo parts: phoneme distance: 3 phonemes distances added togetherLip shape distance for th

16、e overlap portion of triphone videoWeighted add together two partBackground video generationBackground is a video sequence when the virtual character spoke something elseSimilarity measurement of backgroundSelect “standard frame”The frame with maximal number of frames similar to itFilter out the frames with jerkinessStitch the time-aligned result to background facesWrite back with a maskTransform the synthesized lip to the background faceMask image for write-back operationOriginal background fram