内容大会现场emotionally intelligent robotics

1、Emotionally Intelligent HCI and RoboticsZhengyou Zhang Distinguished Scientist Director, Tencent Robotics X zhengyouEvolution of Human-Computer InteractionHmmm,you might be tooemotional Virtual ReceptionistExample: Magic MirrorMicrosoft has developed a mirror that can read your emotionsRecognize and

2、 greet usersRead their emotionsDisplay the weather, time andother information.Motivations Understanding human emotion is the holy grail for human- computer interaction By combining it with speech and gesture understanding, computers will one day be able to communicate with human naturally Wide array

3、 of applicationsAffect-awarealAutism interventionHonest signalAffect-awaregame developmentassistant/companion devicesEmotion Intelligence EI is vital to make machine behave more like human Four branches of EI (Salovey and Mayer) Perceiving emotions understanding nonverbal signals Reasoning with emot

4、ions using emotions to prioritize attention Understanding emotions interpret the cause of emotion Managing emotions regulating and responding to emotions Computer EI is still in its infancyEmotion API for MCSAutisminterven-tionCustomerserviceAdvertisement/marketingalassistantMeetingdynamicsAdaptiveg

5、amingApplicationsEmotion APIsMCSOther sensors(heart rate, gps, eda, etc.)InputImageDepthAudioTextBasic EmotionsMicrosoft- Internal OnlyFacial Action Coding System (FACS)Mapping Between Basic Emotions & FACS1)2)Neutral BaselineExpressionless face1)2)3)Muscles around the eyes tightened“Crows feet”

6、 wrinkles around the eyesCheeks raised4)Lip corners raised diagonally (s)1)2)3)Eyebrow raisedEyelids pulled up (eyes widened)Mouth OpenBasic Emotion Types1)2)3)Inner corners of eyebrow raisedEyelids looseLip corners pulled down (frown)1)2)Eyebrows pulled down and togetherUpper & lower lids pulle

7、d up (glaring)Margin of lips pulled inLips may be tightened (optional)3)4)1)2)Eyes neutralLip corner raised and pulled back on only oneside of the faceBasic Emotion Types1)2)3)4)Eyebrows pulled downNose wrinkled Upper lip raised Lips loose1)2)3)Eyebrows pulled up and togetherUpper eyelids pulled up

8、Mouth stretched20 Years Ago Gabor WaveletsTwo-Layer PerceptronResultExamplesSensitivity AnalysisData Reduction Deleted 12 less informative points:16, 17, 18, 19, 20, 21,23, 24, 30, 13, 29, and3.Significance of Image Scalesk=1 (lowest res),5 (highest res)Major ChallengesInput data is unrestricted A s

9、tress test for face detection/alignment Huge variation in pose, lighting, skin tone, etc. Few existing work on emotion recognition in the wildPeople dont always agree on the corresponding emotion A single face can exhibit multiple emotions For subtle emotion, still image doesnt have enough informati

10、on (no temporal info)Most data available online are biased toward happy and neutralOur Approach A deep learning based approach24244812126densedensedense565MaxPooling8128128128MaxPooling64641024102448 With various data augmentation Affine transform, lighting augmentation, multi-cropping, voting, etc.

11、3123312333243324Data CollectionStart with FER 2013Web crawled + human labeling 48x48 image resolution28709 training examples3589 validation examples3590 test examples Very noisy dataTrain DWithout data augmentation65.07%With data augmentation 71.73%A Glimpse on Fear CategoryCorrect PredictionWrong P

12、redictionsadangrysurprisesadangrysadhappyangrysadhappyhappyneutraldisgustsurprisesadhappyangrysurprisesadangryData CollectionCrawled 4.5m images with emotional keywords166 emotional adjectives230 celebrity names, 100 popular first names, 166 people related wordsFace detection Active learningUse Dto

13、select confusing facial images for taggingSelf-paced learningUse Dto expand training data based on classification resultsRandomly sampledBiased towards rare emotion typesTagging: FACS vs. Basic EmotionsUse FACS (Facial Action Coding System)More accurate and less subjective.Easy expand to more emotio

14、ns.Cons: Expensive and require a certified tagger.Appearance based emotionCheap and doesnt require a certified tagger. Cons: Very noisy.Crowd Source TaggingTaggers are forced to choose one emotion out of 8, or tag the face image as “unknown”We started with at least 2 taggers agree and up to 5 tagger

15、s.Quality was very bad specially with subtle emotions.We retagged all our data with 10 taggers.Quality improved drastically (detailed next).How many taggers to we need?Numberoftaggers1009080706050403020100012345NUMBER OF TAGGERS678910AGREEMENT PERCENTAGEOld versus new labelEmotion Probability Distri

16、butionMajority Voting (MV)Each face is associated with one emotion, the one that has the majority vote.Multi-Label Learning (ML)All emotions above certain threshold are treated as valid emotion.Probabilistic Drawing (PLD)During training draw the target emotion according to its probability.Cross-entr

17、opy (CEL)Learn the actual probability distribution.Emotion Probability Distribution Training resultSchemesTrialsAccuracy12345MV83.60%84.89%83.15%83.39%84.23%83.85±0.63%ML83.69%83.63%83.63%84.62%84.08%83.97±0.36%PLD85.43%84.65%85.34%85.01%84.50%84.99±0.37%CEL85.01%84.59%84.32%84.80%84.

18、86%84.72±0.24%Final Data Set FER + In-houseTrainValidTestNeutral55,1801,1514,396Happiness26,2709041,801Surprise15,421422725Sad11,221418308Angry14,063305843Disgust3,3721987Fear5,44292198Contempt5,3292426Total136,2983,3358,384Performance Nov 2015 release 80.98% on test set Nov 2015 network on bet

19、ter tags 83.66% on test set Our latest deeper network 85.26% on test setNeuHapSurSadAngDisFeaConNeu88.81%3.09%4.03%2.00%1.23%0.11%0.66%0.07%Hap10.83%82.62%2.94%1.05%2.28%0.06%0.11%0.11%Sur7.31%1.66%83.03%0.14%2.48%0.00%4.97%0.41%Sad18.83%1.30%0.97%70.45%2.27%0.97%5.19%0.00%Ang6.05%1.66%2.02%0.95%88.

20、14%0.36%0.59%0.24%Dis18.39%2.30%3.45%2.30%19.54%51.72%0.00%2.30%Fea4.04%1.01%19.19%4.04%3.03%0.00%68.69%0.00%Con26.92%3.85%3.85%3.85%7.69%3.85%0.00%50.00%Some Examples“ Kim Kardashian, with apeculiar 12 percent tint of potential happiness.”More Examples“Pretty awesome that it detectedthe underlying

21、emotion.”“The face of pure happiness.”More Examples“According to Microsoft's Emotion API, Sidney Crosby was rather angry aboutscoring the goal that won Canada a gold medal at 2010's Olympics in Vancouver.”Emotion fromI want to try it Emotion APIImage andDetects happiness, sadness, surprise,

22、anger, fear, contempt, disgust or neutral.Returns score for each emotion, sum to oneREST API, samples inCurl, C#, Java, JavaScript, Object C, PHP, Python, RubyDemo:I want to build something similar Augmented Emotion Datasetand CodeRelabeled FER 2013 data set10 labels per imageCNTK source code for tr

23、aining and testingin PythonSame code we use for creatingEmotion APIReferences Z. Zhang, “Feature-Based Facial Expression Recognition: Sensitivity Analysis and Experiments With a Multi-LayerPerceptron”, International Journal of Pattern Recognition and Artificial Intelligence, Vol.13, No.6, pages 893-

24、911, 1999. E. Barsoum, C. Zhang, C. Canton Ferrer, and Z. Zhang, “Training Deep Networks for Facial Expression Recognition with Crowd- Sourced Label Distribution", in Proc. 18th ACM International Conference on Multimodal Interaction (ICMI), Tokyo, Japan, November 12-16, 2016.AcknowledgmentCha ZhangEmad Barsoum Anna Roth Chris ThrasherCristian Canton Ferrer Oliver WhyteWhy Tencent Robotics X? 腾讯的：、，都是以人 发展人技术是以人的延续以人 在腾讯“18岁”生日时说，腾讯迎来了“”，需要担当的责任 中国的一个迫切的问题是的化; 发展人是一个应对化的自然选择 响应工业4.0布局，帮助企业从数字化、智能化提升智能制造，人是必不可少的