Examining the Effect of Crossmodal Perception and Implicit Memory on Gestural Input Behaviour

Abstract

This thesis explores the influence of crossmodal perception and implicit memory on gestural input behaviour in Human-Computer Interaction (HCI). Although there has been previous research into gesture modelling and evaluation for screen-based interaction system, there is very limited research on how gesture inputs, which performed in a different spatial-temporal scale, can be enhanced by a perceptual phenomenon, crossmodal correspondences (CCs). Informed by the theoretical account of embodied cognition, this thesis seeks to contribute to unveiling the potential of leveraging embodied experience on CCs in gesture-related interactions with two focuses: the relationship between perception and action, and between memory and action. The sub-question 1 relates to the first focus. Two experiments in study 1 addressed this question by investigating the modulation effect of CCs on continuous arm-based gestural input. The sub-question 2 relates to the second focus. Three experiments in study 2 addressed this question by exploring whether and how implicit memory influence consecutive hand-based inputs. The thesis then turn back to the main research question by combining the two foci. The purpose is to deepen the understand on how interaction task and the gestural motion quality can be improved. Two experiments in study 3 extend the experimental approach of study 2 and examine augmented crossmodal feedback and implicit memory priming with continuous handbased and arm-based gestural input respectively. At this point, the results of the three studies highlighted the spatial characteristic of the crossmodal information employed, as well as the spatial accuracy of gestural performance. Finally, a further study was conducted to evaluate the temporal performance of both the hand-based and arm-based gestural input behaviours. The thesis concludes by drawing together the results of the studies into a series of implications relating to the design and application of embodied interaction. This work contributes to the area of Multimodal Interaction and HCI by providing empirical evidence of how embodied perceptual experience can be leveraged to enhance gestural input quality and interaction performance. It also provides a set of design implications and an application outlook to aid design and inspire future research.