Bringing Musicality to Movement Sonification: Design and Evaluation of An Auditory Swimming Coach First Author
Second Author
Gabriela Seibert
Daniel Hug
Oberer Hofacker 300
University of Applied Sciences and Arts
5044 Schlossrued,
Northwestern Switzerland
Switzerland
School for Teacher Education
[email protected]
Chair for Music Education Obere Sternengasse 7 4502 Solothurn, Switzerland
[email protected]
Abstract In this paper we describe a novel approach to the sonification of crawl swim movement. The design method integrates task and data analysis from a sport science perspective with subjective experience of swimmers and swimming coaches, and strongly relies on the skills of musicians in order to define the basic sonic design. We report on the design process, and on the implementation and evaluation of a first prototype.
Keywords Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that
Movement Sonification, Swimming Sport Training, Musical Improvisation
copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on
ACM Classification Keywords
the first page. Copyrights for components of this work owned by
H.5.2 [User Interfaces]: Auditory (non-speech) feedback; H.5.2 [User Interfaces]: Evaluation/methodology; H.5.2 [User Interfaces]: Prototyping; J.5 [Arts and Humanities]: Performing arts
others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected].
General Terms AM '13, September 18 - 20 2013, Piteå, Sweden Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-2659-9/13/09…$15.00. http://dx.doi.org/10.1145/2544114.2544127
Design, Human Factors
Introduction In swim practice, the subjective imagination of movement is the basis for the actual execution of the swimming movement, for instance for the crawl stroke. Usually, the training of this mental representation relies on visual and verbal information [5]. But auditory movement information offers several advantages, in particular for crawl swimming. Hermann et al. state that “sound is accessible without demanding visual attention (which would be difficult underwater), our auditory perception has a high temporal resolution (…), and we are highly sensitive to rhythms and changes of rhythms, and these patterns occur frequently in repetitive coordinated body movements” [6, p. 60].
Past work In a preceding pilot project, an optimal movement of the crawl arm stroke was sonified. The aim was to create and evaluate a training aid which could be used by everyone, does not require an extensive technical setup and would provide real-time orientation (from the pre-recorded tracks) for the execution of the swimming movement. For this reason, a non-interactive sonification was chosen. The movement of the hand underwater was segmented in four phases. Using a parameter mapping sonification (PMSon), every phase was represented with a sound, matching the predefined duration of the phase. The evaluation with swimmers showed, that the sonification was intelligible and helpful, but there was a general dissatisfaction with the sound itself, from an aesthetic point of view [10]. This limited the usefulness of the sonification, because not only the information, but also a motivation for the movement is necessary for the training and execution of movement [5]. For this reason, the aim of the project described here was to find a suitable strategy to
create a sonification which integrates the informational aspect with the aesthetic dimension.
Method Barras and Vickers propose to collaborate with skilled experts in aesthetic thinking and practice [2]. Following this suggestion, and inspired by the video artwork “Forty-eight to Sixteen”, where the heart rate, breathing, and cadence of a cyclist are interpreted by a cellist [6], we decided to integrate musicians in the design process. For this purpose we established a process of ongoing interaction between sports science and musical practice. As previous research [7] revealed advantages of using improvisational and performative methods for designing auditory displays, our aim was to provide the musicians with a basis for an improvised interpretation. This way, their expertise in rich expression through sound could be brought to fruition in sonification. Based on the three dimensions of sonification, the movement properties, their mapping and the sound properties [4] the following steps have been used in the design process: Movement Properties In order to decide, which movement information should be selected for sonification, qualitative, semisstructured expert interviews with five swimming coaches were conducted. In order to complement the information with a more evocative perspective, the interviews also covered aspects of the subjective experience of crawl swimming. Based on the the results, several biomechanical and psychomotor dimensions were described (task-dimensions of the sonification). Subsequently, the biomechanical data
domain was defined, and psychomotor metaphors, based on the descriptions of the swimming experience, were derived. This was done in collaboration with the swimming coach expert Gunther Frank. The resulting data were visualized on a temporal scale, using the biomechanical data from "Swimming Fastest" [8] and synchronized with the video analysis of swimming by Grant Hackett [1] (Figure 1). The visualization served as a visual score and improvisation guideline for the musicians. It represents the data of a crawl arm stroke using seven parameters for the arm stroke and three parameters for the remaining body movement. A cursor follows the movement from left to right, indicating the position in the video together with the position on the visualization. In order to give
musicians enough time for their interpretation of the movement properties, the video was played back with 15-fold speed in this first trial. Mapping The mapping builds on the interaction between musical interpretation and data based on scientific and subjective analysis of the swimming process. In order to facilitate the musical interpretation of the visualization, it was first explained to the musicians. A rough guide for the improvisation was provided by the researcher. Then the musicians interpreted and discussed the experiential metaphors and discuss their improvisational approach.
The colored rectangle in the center represents (1) stroke pattern phases (entry/stretch, downsweep, insweep, upsweep, release, recovery), as well as a graph of the hand speed (2), points for the leg kick, (8) stars indicating the body roll (9) and symbols for breathing points (10). The curves below indicate the horizontal (3) and the vertical (4) position of the hand. The three curves at the bottom represent the angle progression of wrist (5), elbow joint (6), and shoulder joint (7).
Figure 1. Improvisation guideline. The movement is visible in the video on top and a synchronized cursor line runs across the visualization below
4
Sound Properties Based on this concept and the visualization, five musicians from the Free Jazz combo from Shanti Music performed in a recorded improvisation. One session focused on the arm stroke and another session on the remaining body movement. This enabled a more flexible positioning and layering of the two components of the overall swimming movement. The stroke pattern phases were interpreted with sounds which were inspired by the experiential metaphors provided by the swimming coaches. The hand velocity was interpreted as tempo change by all instruments. The horizontal position of the hand was illustrated by a volume change of the drums and the vertical position by a change of pitch of the guitar. The angular change of the wrist (bass guitar), elbow joint (saxophone 1) and shoulder joint (saxophone 2) were interpreted as change of intensity. In the second recording for the remaining body movement, leg strokes were associate with left and right toms and body roll with the tilting of a rain stick, and breathing with a hit on an Urdu. The resulting music begins with a calm sound of bass guitar, guitar, saxophones and drums, then an accelerating bass pizzicato and a "bending" sound of saxophone 1, which are followed by several increases in intensity by both saxophones. The wave-like noise of the rain stick joins all elements into an overall movement. The music has no steady beat or rhythm, rather a free ebb and flow. The recordings were then mixed on the left and right channel of a stereo file, synchronous to the crawl movement of the right and left arm. The second recording for the overall body movement was layered
on top. This resulted in a composite sound file of 30 seconds duration. In order to synchronize with the realistic crawl stroke, the recording was accelerated (without pitch change) to an overall duration between 1.7 and 2.7 seconds, with ten intermediary durations, each 0.1 second apart, and looped. This allowed swimmers to play the sound back in synchronicity with their target stroke rate. It was assumed, that essential characteristics of the sound would be preserved, despite the increase of speed. Test and Evaluation The evaluation of this first prototype was done using semi-structured interviews, combined with five point Likert scale ratings. The aim was not only to get substantial feedback for future development, but also to preserve the subjective experience of the sonification. Tests and interviews were carried out with five coaches and five swimmers in an indoor swimming pool. Sex and musical experience was equally distributed, age range was from 22 to 76 and both amateur and professional swimmers contributed. The interview covered the areas "listening", "understanding" and "application". For the sections "listening" and "understanding", the participants first listened to a playback of the sonification. For the section "application", they first swam 200m, listening to the sonification using a waterproof mp3 player.
Results All participants mentioned that rhythm was an essential movement property of crawl swimming. Thus in particular the sonification of hand speed and movement phases worked well and should be maintained. Also the sonification of the leg kick and the various joint angles (which indicate muscle contraction) were appreciated.
5
The mapping was perceived as meaningful by most participants. However, several participants judged the sonification of the movement phases to be too hectic and the representation of movement properties to be too unclear and imprecise. The sound of the music was appropriately associated with water, waves and gliding. The interleaving of tension and relaxation in the musical interpretation raised curiosity and joy. On the other hand, the monotony of the repetition was criticized by some participants. Three participants stated that they would prefer a tune or song with rhythm. On the other hand, the remaining seven participants stated that it was exactly the monotony that supported them in an effective technical and mental training and the internalization of movement. The rhythm and speed was described as being well defined but too dominant. According to one participant, rhythm should mainly serve to underscore or accompany the movement. Regarding the application in a swimming task the musical sonification was rated as helpful by nine participants, not disturbing by six participants and aesthetically satisfying by six participants. More than half of the participants judged the music to have a positive influence on mood, motivation, avoidance of boredom and stated that it may help to cope with excessive strain. The rate of advancing, the movement rhythm and precision were rated as clearly improved by all participants. Participants stated that the ideal application of the musical sonification could be as rhythm support in technical training on the one hand, and as motivation aid in endurance training on the other hand. The idea
emerged to use this kind of sonification in the planning of training and support training with individually composed programs using a musical representation of various technical exercises, key optimization points, training duration, training intensity, training volume and training density.
Conclusion and Future Work Overall, the method of integrating musical inspiration and expertise in the process of generating a sonification was successful. The creation of a visual improvisation guideline that would serve as foundation for developing an improvisation concept together with musicians and the resulting actual improvisation session worked very well and the result were greatly improved compared to the previous version which relied on simple mappings between parameters and simple sound dimensions such as pitch or instrument. Thus, the main goal of creating a sonification which satisfies not only functionally but also aesthetically was reached. Both the design method and the resulting sonification have potential for improvements. A future version should cover fewer and more relevant movement dimensions, but be more exact, clear and less hectic. It will be necessary to consider knowledge not only about the appropriate use of simultaneous information streams in the sonification but also the useful amount of simultaneous dimensions of conscious control of movement execution. In terms of involving musicians in the process, the visual improvisation guideline was criticized for being too imprecise and also the strong temporal compression was an issue. A challenge is to find a suitable duration for the sonification of one stroke
6
which gives the musicians enough "expressive" space, but also does not require excessive temporal compression for the matching to the actual stroke frequency. One approach to deal with temporal compression might be derived from the concept of "Spearcons" [9], which are based on spoken phrases that are time compressed until they are not recognized as speech, however the approach would have to be adapted to musical requirements. In order to optimize the functional benefit, there needs to be additional guidance for the improvisation session, incorporating training theory and body mechanics. Still, a rather free improvisation is highly recommendable, in order to explore and develop a "sonic language" which relates to the emotional and metaphoric qualities of the swimming experience. It might be also interesting to investigate the outcome of the musical interpretation when collaborating with musicians from another background than (Free)Jazz, e.g. classical music, electronica, or even non-western musical systems such as Raga. Because of the qualitative nature of the evaluation we cannot generalize the results regarding aesthetic and functional guidelines for swimming sonifications. But the investigation resulted in valuable rich data rooted in high level of expertise and the reports of personal experience helps to develop solutions that take into account the subjective and emotional dimension of sport training. Further development improve the creative method, further integrate the musicians, but also provide more precise direction for the improvisation and reduce the impact of the temporal compression. The resulting sonification will be evaluated both qualitatively and
quantitatively and will address training effectiveness over a longer time period.
Acknowledgements We thank Shanti Music for their outstanding and essential musical contribution, Gunter Frank for valuable feedback and guidance and all interviewed coaches and swimmers.
References [1] 1500m Freestyle men final, part 1 (2001). http://youtu.be/D7ykhhRcKDE . [2] Barrass, S., Vickers, P. Sonification design and aesthetics. In Hermann, T., Hunt, A., Neuhoff, J.G, The Sonification Handbook (2011), 145-172. [3] Forty-eight-to sixteen. http://brianhouse.net/works/forty_eight_to_sixteen/ . [4] Grond, F., Berger, J. Parameter Mapping Sonification. In Hermann, T., Hunt, A., Neuhoff, J.G, The Sonification Handbook (2011), 363-398. [5] Grosser, M., Neumaier, A. Techniktraining: Theorie und Praxis aller Sportarten. In BLV Sportwissen (1982). [6] Hermann, T., Ungerechts, B., Toussain, H., Grote, M. Sonification of pressure change in swimming for analysis and optimization. In Proceedings of ICAD (2012), 60-67. [7] Hug, D. Performativity in Design and Evaluation of Sounding Interactive Commodities. In Proceedings of Audio Mostly 2010 (2010). [8] Maglischo, E.W. Swimming Fastest. Human Kinetics (2003). [9] McGookin, D., Brewster, S. Earcons. In Hermann, T., Hunt, A., Neuhoff, J.G, The Sonification Handbook (2011), 339-362. [10] Seibert, G. Sonifikation. Kraularmzugbewegungsmuster. (unpublished) PH FHNW, Institut Sekundarstufe, Solothurn/Brugg (2012).
