Games 4 VRains: Affective Gaming for Working Memory Training in Virtual Reality

Abstract

The explosion of Virtual Reality (VR) in the last few years, thanks to the introduction of affordable Head-Mounted Displays (HMD), has increased the interest in this technology for research. One of the main research areas using VR is the field of cognitive and physical rehabilitation or training. Although it is in early stages, many researchers have shown the positive effects of the higher levels of immersion, often reported in VR, on cognitive skills. Video games have also been used for cognitive training due to their capacity to engage and motivate players. Recent findings have demonstrated that by adapting the game to the player’s performance, real cognitive benefits can be achieved as the adaptation offers a personalised cognitive training program. However, this adaptation normally considers just performance metrics and ignores other crucial aspects like the player’s affective states or experience. Arousal and valence have generally been shown to enhance the subjects’ cognitive skills and thus should also be considered when adapting a game for cognitive training. Following these findings, this thesis investigates the effects of affect and performance-based adaptation of a VR video game on player’s working memory (WM) performance. An initial pilot study explores suitable ways of measuring player’s arousal and valence levels through physiological and behavioural cues. In a second study, the effects of immersion, arousal and valence on player’s WM performance in Desktop and VR gaming are examined. The results of this study show that players in an optimal affective state can significantly improve their WM performance, supporting the incorporation of affective metrics in the adaptation engine. Thus, an adaptation engine was developed, implemented and tested to automatically adjust the game’s difficulty level depending on the player’s performance and the detected affective state. Two machine learning algorithms in the adaptation engine recognise and classify player’s arousal and valence levels using physiological and behavioural features for adaptive decision making. Across the three studies presented, this thesis makes the following novel contributions. It shows that, i) VR is a suitable medium for cognitive training since the elicited high levels of immersion have a positive effect on players’ WM performance, ii) positive affective states help subjects to achieve a better WM performance, and ii) difficulty adaptation is more beneficial for subjects with low WM capacity. During this process, it also provides a new methodology for affect recognition in VR gaming and a novel adaptation engine compounded by affect and performance metrics. Therefore, this work proposes that gamebased cognitive training would be improved by VR, especially by the use of affective and performance metrics for dynamic adaption, resulting in a highly personalised and more effective training experience.