智慧型家庭网路之技术与应用.ppt

1、1,智慧型家庭網路之技術與應用,Professor Yau-Hwang Kuo Director Center for Research of E-life Digital Technology (CREDIT) National Cheng Kung University Tainan, Taiwan,Outline,Introduction Structure of Smart Home Network Realization of Device Cameras; Speakers,Home Comm. Gateway; Home Perception Server; Home Media

2、 Center,Wireless A/V Streaming Appliances,Mobile Agent Platform,Agent Layer,Emotion / Semantics / Behavior / Intention Understanding,Corpus of Knowledge (Ontology),Inference Engine,Natural Language Processing (text, spoken),Cognition / Affection Layer,Vision (gesture),Device signal analysis Device-f

3、ree type: speech interaction; vision monitoring. Seamless handoff and transcoding for ubiquitous service following Roaming path tracking and prediction,Agent-based Middleware: appliance collaboration management,Collaboration among homogeneous appliances: data fusion, task migration. Collaboration am

4、ong heterogeneous appliances: multi-modal HCI. Scheduling, concurrency control & synchronization of collaborative tasks. Self-organization for service deployment,Agent-based Middleware: interoperability management,Device bridge Protocol bridge Transcryption Transcoding Content translation & adaptati

5、on,Agent-based Middleware: remote access management,Remote service deployment remote service access remote service management auto-configuration service re-direction service aggregation UI remoting,Agent-based Middleware: other management functions,load management: Client-server load partition Serve

6、r load sharing Load scheduling of appliance farm availability management Fault tolerance Just-in-time activation of appliances service quality management,Affective Speech Conversation,Emotional Speech Synthesis,Behavior Understanding by Vision,High-Level behavior understanding from videos State Mach

7、ine Human Activity Recognition Two-Stage recognition process Accident/Abnormal behavior detection Context & domain knowledge Combination,System Architecture,Method Activity Recognition,Activity Recognition Level 1 - postures Posture Sequence Level 2 motion/history History Map Matching,Method Behavio

8、r understanding,Behavior Normal behavior State Machine Activity + Contexts Abnormal behavior Normal behavior + domain knowledge Accident Unreasonable activity + domain knowledge,Facial Expression Analysis,Face Acquisition,Acquisition,Segmentation,Facial Feature Extraction,Deformation Extraction,Moti

9、on Extraction,Representation,Facial Expression Classification,Recognition,Key frame Selection,YCbCr Color space,Eye Region,Mouth Region,Region Of Interest,Eye Points,Mouth Points,Displacement Vectors,Fuzzy Neural Network,Invariant Moments,Optical Flow,Key Frame,Image Sequence,Integrated Perception:f

10、uzzification of reference perceptual models,Manipulate all kinds of perception in a uniform process to ease the perceptual integration. Due to high vagueness of perception, fuzzy logic based approach is a good choice to establish the reference models of perception. The reference models which fuzzify

11、 perceptual attributes and perceptual decision subspaces will be embedded into the integrated perception model.,FL-based Acoustic Reference Model for Emotion Recognition,feature extraction,AAU1 model,SVM clustering for emotion 1,SVM clustering for emotion 2,SVM clustering for emotion V,speech corpus

12、,AAU2 model,AAUS model,fuzzification of acoustic features (AFs) and construction of acoustic action units (AAUs),FL-based Acoustic Reference Model for Emotion Recognition (cont.),Adopt SVM clustering approach in the subspace of each emotion type to gather the clusters of acoustic training patterns.

13、Inspect all produced SVM clusters in the whole feature space and merge the highly overlapped clusters. Each cluster is modeled as an AAU represented with its fuzzy cluster center where each feature is a fuzzy set whose membership function is determined by the least-square curve fitting approach on t

14、he feature values of training samples included in the cluster.,FL-based Acoustic Reference Model for Emotion Recognition (cont.),The mapping between AAUs and emotion types is dependent on the SVM clustering result of each emotion type. Each emotion type is associated with a set of clusters of acoust

15、ic samples. The weight of each cluster is determined by the ratio of the number of samples it contains with respect to the total amount of samples of the same emotion.,FL-based Facial Reference Model for Emotion Recognition,graphical head model,morphological process to simulate AUs,FACS AUs identifi

16、cation process,feature points (FPs) extraction process,fuzzy logic based reference model for FACS,correspondence,Membership grade,Membership grade,FP1 value,FP2 value,FAU1,FAU2,FAUi,FAUj,FAUk,FL-based Facial Reference Model for Emotion Recognition (cont.),Intend to construct a computational referenc

17、e model for FACS action units based on the measurable features of facial expression. An approach similar to the construction of acoustic reference model is adopted. The training samples are generated from a generic head model with necessary morphological manipulation.,FL-based Facial Reference Model

18、 for Emotion Recognition (cont.),The membership functions will be determined by the least-square curve fitting approach according to the sample patterns produced from the morphological process. Each AU may just represent a partial facial expression and relate to more than one emotion.,FAU1,FAU2,FAUK

19、,AAU1,AAU2,AAUS,FP1,FPn,AF1,AFm,emotion type layer,representative concept layer,scaled feature layer,primary feature layer,Face Features Expression,Acoustic Features,Fuzzy Neural Network for Integrated Emotion Recognition,Fear,Anger,Surprise,Fear,Anger,Surprise,Fuzzy group decision process, group le

20、vel of agreement,Fuzzy Neural Network for Integrated Emotion Recognition (cont.),All kinds of perceptual information are fused by the FNN model to realize emotion recognition. Each appliance will have an instance of the corresponding FNN to join the emotion recognition job. A two-layered (emotion ty

21、pe & concept layers) BP learning algorithm is adopted by using the training samples in constructing reference models. The fuzzy group decision process does not join the learning. Scaling input value to 0,1 in the second layer is realized by the membership function of the corresponding fuzzy set.,Fuz

22、zy Neural Network for Integrated Emotion Recognition (cont.),The links between AUs and scaled features are not fully connected. The FAU/AAU nodes realize normalized weighted sum for the membership grades of input features weighted by their respective link strength. Each emotion type node determines

23、output value by the normalized weighted sum of its inputs from the representative concept layer.,Cognition Layer:understanding and response,Understand the semantics of multi-modal expression. Classify and recognize the intention/ need/emotion of semantic expression. Summarize the semantics of multi-

24、modal expression according to classified result.,Cognition Layer:understanding and response (cont.),Predict the user behavior sequence according to the classified result. Schedule the response sequence according to the prediction result. Determine the instant response.,Stimulus,Perception,Cognition,

25、spoken language,gesture,face expression,physiological signals,text,Speech Processing,Vision Processing,Signal Processing,Video Processing,Speech Processing,Application Control,Response,Semantic Feature Extraction,Conceptualization,Event Detector (Neural Network- based Approach),Emotion Recognition,E

26、motion Episode Discovery,Ontology,Personal Event / Emotion Log,Contextual Rules,Stimulus Semantic Summary Extraction,User Behavior Prediction (Episode-based Approach),Response Scheduling,Instant Response Determination,Stimulus Response Templates,Event Sequence Case base,Emotion Sequence Case base,Semantic Expression,Emotion Attributes,Features,Concepts,Events,Semantic Summary,Emotion Episodes,Prediction Result,Response,Emotion Types,Response Roadmap,Smart Home Services,nomadic content services health care by integrated perception smart hom