| dc.description.abstract | Artists and researchers have long theorized and evaluated connections between
sound and image in the context of musical performance. In our investigation,
we introduce music information retrieval (MIR) techniques to the practice of live
sound visualization and utilize computer-generated graphics to create aesthetic
representations of music signals in real time. This thesis presents research that
assesses design requirements for live audiovisual practice and evaluates different
sound and image interaction systems. We propose a visualization method
based on automated music analysis and multitrack audio to provide fine controls
for audio-to-visual mapping and to support creative practice. We adopted a
user-centered design approach informed by a meta-analysis of user studies exploring
contemporary methods of live visual performance. We then conducted online
surveys collecting general and specialist knowledge about audiovisual practices,
multitrack audio, and audio feature extraction from over 50 practitioners. We performed
research through design (RtD) and developed four audiovisual artifacts
to test different audiovisual paradigms according to user interaction with audio
data, mapping strategies, expression, and affordances. This helped us identify features
and limitations of audiovisual models for live performance. Our final prototype
(FEATUR.UX.AV) enables users to compose live visuals driven by audio
features extracted on multiple instrumental audio stems. We conducted an experiment
with 22 audiovisual performers to assess visualization methods in different
audio input (multitrack, final mix) and audio feature (raw audio, content-based
audio features) conditions. We used Human Computer Interaction (HCI) frameworks
to assess usability, hedonic experience, preference, and value as a creativity
support tool. In addition to established frameworks, we used qualitative methods
to analyze reflective feedback from open answer questions related to aspects of
user experience. This evaluation helped us gain insight into the nuances of user
experience and highlight advantages and drawbacks of multitrack audio and audio
content-analysis for live audiovisual practice. | en_US |
