Melodic sonification of movement as concurrent augmented feedback enhances learning in a bimanual task John Dyer¹, Dr. Paul Stapleton² and Dr. Matthew Rodger¹ ¹School of Psychology, ²Sonic Arts Research Centre
Sonification: concurrent auditory feedback
Results
Concurrent augmented feedback (e.g. graphs, Lissajous displays) which is generated and displayed live, usually enhances performance on motor tasks, but learners can become dependent on it. This is not ideal because feedback may not always be available.
We compare participants who heard sound in the demo alone (SoundDemo), those who had their movements sonified as well (Sonification) and those who learned the task entirely without sound (NoSound).
However, sound as feedback may not be subject to the same effect (Ronsse et al., 2011). Sonification is the transformation of data into sound. In this case, it is used to convey task-relevant information about motor performance.
We look at both performance and learning (as measured on no-feedback retention tests after 5 minutes and 24 hours). The measure used is deviation from the ideal (4:3) ratio
With sonification, a motor task can be thought of as a musical performance. Perform the task correctly and produce the correct sound by doing so. We use optical motion capture to measure performance and capture movement data for sound synthesis.
Task to be learned Rhythmic bimanual coordination tasks are difficult to produce without extensive practice (Summers et al., 1993). Performance is highly dependent on being able to detect higher-order information about the relationship between the hands – this is normally provided via a feedback display.
We designed a shape tracing task in which the goal is to move with a 4:3 intermanual timing ratio. We sonify a demonstration of the task and the learner’s movement so that fingertip arrivals at shape corners trigger tones. These form a melody.
Conclusions and next steps Good performance is not dependent on the presence of concurrent feedback after learning. Next we will try to extend the benefit of sonification by emphasising the musicality of the task. We hope to produce longer-lasting retention.
Custom-built shape-tracing apparatus and one of two motion-tracked gloves
Email me at
[email protected] Or find me on Twitter @JFDyer
References: Ronsse, R., Puttemans, V., Coxon, J. P., Goble, D. J., Wagemans, J., Wenderoth, N., & Swinnen, S. P. (2011). Motor learning with augmented feedback: modality-dependent behavioral and neural consequences. Cerebral Cortex, 21(6), 1283–94. Summers, J. J., Rosenbaum, D. A., Burns, B. D., & Ford, S. K. (1993). Production of polyrhythms. Journal of Experimental Psychology: Human Perception and Performance, 19(2), 416–428.
