In this paper, we explore the use of computer vision techniques to analyze students' moods during one-on-one teaching interactions. The eventual goal is to create automated tutoring systems that are sensitive to the student's mood and affective state. We find that the problem of accurately determining a child's mood from a single video frame is surprisingly difficult, even for humans. However when the system is allowed to make decisions based on information from 10 to 30 seconds of video, excellent performance may be obtained.
JF - 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011) PB - IEEE CY - Santa Barbara, CA SN - 978-1-4244-9140-7 ER - TY - CONF T1 - The computer expression recognition toolbox (CERT) T2 - 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011) Y1 - 2011 A1 - Littlewort, G. A1 - Whitehill, J. A1 - Wu, T. A1 - Fasel, I. A1 - Frank, M. A1 - Movellan, J. A1 - Bartlett, M. KW - 3D orientation KW - Accuracy KW - automatic real-time facial expression recognition KW - CERT KW - computer expression recognition toolbox KW - Detectors KW - dual core laptop KW - Emotion recognition KW - Encoding KW - extended Cohn-Kanade KW - Face KW - face recognition KW - facial action unit coding system KW - facial expression dataset KW - Facial features KW - FACS KW - Gold KW - Image coding KW - software tool KW - software tools KW - two-alternative forced choice task AB -We present the Computer Expression Recognition Toolbox (CERT), a software tool for fully automatic real-time facial expression recognition, and officially release it for free academic use. CERT can automatically code the intensity of 19 different facial actions from the Facial Action Unit Coding System (FACS) and 6 different prototypical facial expressions. It also estimates the locations of 10 facial features as well as the 3-D orientation (yaw, pitch, roll) of the head. On a database of posed facial expressions, Extended Cohn-Kanade (CK+[1]), CERT achieves an average recognition performance (probability of correctness on a two-alternative forced choice (2AFC) task between one positive and one negative example) of 90.1% when analyzing facial actions. On a spontaneous facial expression dataset, CERT achieves an accuracy of nearly 80%. In a standard dual core laptop, CERT can process 320 × 240 video images in real time at approximately 10 frames per second.
JF - 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011) PB - IEEE CY - Santa Barbara, CA SN - 978-1-4244-9140-7 ER - TY - CONF T1 - Learning to Make Facial Expressions T2 - IEEE 8th International Conference on Development and Learning, 2009. ICDL 2009 Y1 - 2009 A1 - Wu, T. A1 - Butko, N. A1 - Ruvulo, P. A1 - Bartlett, M. A1 - Movellan, J. KW - Actuators KW - Emotion recognition KW - face detection KW - face recognition KW - facial motor parameters KW - Feedback KW - Humans KW - learning (artificial intelligence) KW - Machine Learning KW - Magnetic heads KW - Pediatrics KW - real-time facial expression recognition KW - Robot sensing systems KW - robotic head KW - Robots KW - self-guided learning KW - Servomechanisms KW - Servomotors AB -This paper explores the process of self-guided learning of realistic facial expression production by a robotic head with 31 degrees of freedom. Facial motor parameters were learned using feedback from real-time facial expression recognition from video. The experiments show that the mapping of servos to expressions was learned in under one-hour of training time. We discuss how our work may help illuminate the computational study of how infants learn to make facial expressions.
JF - IEEE 8th International Conference on Development and Learning, 2009. ICDL 2009 PB - IEEE CY - Shanghai SN - 978-1-4244-4117-4 ER - TY - CONF T1 - Auditory mood detection for social and educational robots T2 - IEEE International Conference on Robotics and Automation, 2008. ICRA 2008 Y1 - 2008 A1 - Ruvolo, P. A1 - Fasel, I. A1 - Movellan, J. KW - auditory mood detection KW - Computer vision KW - educational robot KW - Educational robots KW - Emotion recognition KW - emotional speech database KW - face detection KW - hearing KW - interactive robotic application KW - learning (artificial intelligence) KW - Machine Learning KW - Mood Prototypes KW - object recognition KW - Robotics and Automation Robots KW - social mood KW - social robot KW - Speech KW - USA Councils AB -Social robots face the fundamental challenge of detecting and adapting their behavior to the current social mood. For example, robots that assist teachers in early education must choose different behaviors depending on whether the children are crying, laughing, sleeping, or singing songs. Interactive robotic applications require perceptual algorithms that both run in real time and are adaptable to the challenging conditions of daily life. This paper explores a novel approach to auditory mood detection which was born out of our experience immersing social robots in classroom environments. We propose a new set of low-level spectral contrast features that extends a class of features which have proven very successful for object recognition in the modern computer vision literature. Features are selected and combined using machine learning approaches so as to make decisions about the ongoing auditory mood. We demonstrate excellent performance on two standard emotional speech databases (the Berlin Emotional Speech [W. Burkhardt et al., 2005], and the ORATOR dataset [H. Quast, 2001]). In addition we establish strong baseline performance for mood detection on a database collected from a social robot immersed in a classroom of 18-24 months old children [J. Movellan er al., 2007]. This approach operates in real time at little computational cost. It has the potential to greatly enhance the effectiveness of social robots in daily life environments.
JF - IEEE International Conference on Robotics and Automation, 2008. ICRA 2008 PB - IEEE CY - Pasadena, CA SN - 978-1-4244-1646-2 ER - TY - CONF T1 - Automatic cry detection in early childhood education settings T2 - 7th IEEE International Conference on Development and Learning, 2008. ICDL 2008 Y1 - 2008 A1 - Ruvolo, P. A1 - Movellan, J. KW - Acoustic noise KW - auditory moods KW - automatic cry detection KW - behavioural sciences computing KW - Deafness KW - early childhood education settings KW - education KW - Educational robots KW - Emotion recognition KW - human coders KW - Humans KW - learning (artificial intelligence) KW - Machine Learning KW - Mood KW - preschool classrooms KW - Prototypes KW - Robustness KW - Working environment noise AB -We present results on applying a novel machine learning approach for learning auditory moods in natural environments [1] to the problem of detecting crying episodes in preschool classrooms. The resulting system achieved levels of performance approaching that of human coders and also significantly outperformed previous approaches to this problem [2].
JF - 7th IEEE International Conference on Development and Learning, 2008. ICDL 2008 PB - IEEE CY - Monterey, CA SN - 978-1-4244-2661-4 ER -