Personality, Mood and Affect Recognition Based on Neuro-Physiological Signals for Individuals and Groups

Abstract

This thesis falls in the field of affective computing and more specifically addresses the task of implicit recognition of people’s affective factors namely affective states, personality, and mood by means of the analysis of their physiological signals. We present three main contributions to the field. The first one is a novel database designed with a holistic approach for multi-modal research of affective phenomena such as affective states, personality, mood, affective contagion, etc. Our experiments involve participants watching short and long emotional videos either individually or in groups. It also allows the study of the effect that social context has on people’s affective response. The database was collected using wearable sensors for EEG, ECG and GSR recording. The second contribution consists of novel shallow and deep learning methods for subject-independent prediction of affective states based on neurophysiological signals. In the shallow method, features are extracted and then selected based on Fisher’s Linear Discriminant (FLD) and classification is performed with Gaussian Naıve Bayes. We perform feature level fusion and decision level fusion of modalities. The deep learning method consists of a novel parallel CNN and RNN network, intended to combine both the capability of CNNs to deal with spatially related data and the capability of RNNs to deal with temporally related data, for joined valence and arousal prediction. Our shallow method outperforms state of the art methods for affective states recognition. The deep learning method outperforms our shallow method. The third contribution consists of novel shallow and deep learning methods for personal factors prediction using neurophysiological signals. In the shallow method, we use PCA, FLD and Pearson correlation for dimensionality reduction. Linear-SVM was used for classification. The deep learning method uses a multi-task cascaded deep neural network consisting of an RNN applied over the outputs of the affect network for consecutive segments. We include independent costs to regulate the performance of both affective states and personal factors recognition. The shallow method produces statistically significant results. The deep learning method significantly captures the personal factors, showing that it is useful to use affective states as intermediate features to predict personal factors.