Emocolor: Fine-Grained Emotion Recognition from Skin Color Information
Abstract
In everyday human-to-human communication, emotions play a fundamental role. Emotions represent the affective behavior of humans that is multi-modal, subtle, and complex. Previous approaches based on conventional computer vision explicitly used shape information. Modern approaches based on deep learning implicitly exploit all information available in the image, but by their nature make it difficult to assess the contributions of each source of information. In addition, skin color as a unimodal technique to recognize emotions has been explored to recognize only three coarse-grained emotions in valence space.To the best of our knowledge, this work presents the first approach to fine-grained emotion recognition that uses only skin color. Emotions of finer categories such as disgust and fear, happiness, and tenderness were examined as well as multiple feature types, color spaces, and learning models. We provide a baseline for fine-grained classification on three and twelve classes, and for regression tasks on valence and activation dimensions. Regarding regression tasks, the Linear Regression and Ridge Regression models yielded some of the best results. Concerning coarse-grained classification and regression tasks, our results are nearly identical to the state of the art. Our findings were inconclusive with respect to the role of the individual's gender in improving fine-grained emotion recognition. In addition, statistics extracted from temporal windows showed to be more effective than instantaneous values for valence and activation of emotions prediction. An analysis of delays in the time taken by annotators to rate the emotional state of individuals suggests that the emotional changes occur before the emotions are expressed. Finally, we built a dataset of 65 subjects showing spontaneous emotions, their self-report on valence and activation space, and their background information for our project. This work can potentially contribute to research areas such as psychology, cognitive science, neuroscience, behavioral science, and computer science, especially, for applications where subjects are uncooperative and situations where real-time performance is critical.
Subject Area
Engineering|Computer Engineering|Artificial intelligence
Recommended Citation
Jimenez Velasco, Maria Guadalupe, "Emocolor: Fine-Grained Emotion Recognition from Skin Color Information" (2020). ETD Collection for University of Texas, El Paso. AAI28315263.
https://scholarworks.utep.edu/dissertations/AAI28315263