The Smartphone Project An Augmented Dance Performance Leif Oppermann, Clemens Putschli, Constantin Brosda, Oleksandr Lobunets Fraunhofer FIT Schloss Birlinghoven, 53754 Sankt Augustin, Germany {oppermann, putschli, brosda, lobunets}@fit.fraunhofer.de ABSTRACT
The Smartphone Project (TSP) is an interactive danceperformance in a professional setting that exploits the communication channels provided by smartphone-apps as a new material in the dance-theatre domain. We present an account of the experience and its staging. Based on an initial study with 36 participants from the audience, we present results and discuss lessons learned from this project that might guide similar future work. Author Keywords
Audience interaction; dance; theatre; augmented reality; smartphone; material; mixed reality; art; performance; ACM Classification Keywords
H.5.2 Information Interfaces and Presentation: User Interfaces – User-Centered Design; J.5 Computer Applications: Arts and Humanities – Performing Arts INTRODUCTION
The Smartphone Project is a novel interactive danceperformance in a professional setting that explores the smartphone as a material and an artistic medium. Participants, who are paying theatre-goers on an evening out, are encouraged to bring their own mobile device and use it within and as part of the performance. This is facilitated by a dedicated app for Android and iOS that can be installed on their phones and which provides different kinds of interaction between the performers and the audience according to the choreography. Interaction is not limited to only using the app, as audience members are also free to use their phone in any way they like, including answering the phone, taking pictures and videos, and posting them on social media. It is also not strictly required to bring and use one’s own phone, as the piece is carefully Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. CHI 2015, April 18 - 23, 2015, Seoul, Republic of Korea Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-3145-6/15/04…$15.00 http://dx.doi.org/10.1145/2702123.2702538
Fabien Prioville fabien prioville dance company Spichernstrasse 39 40476 Düsseldorf, Germany
[email protected]
designed to be enjoyable even without a phone. It also encourages sharing screens with neighbors, as long as there are enough phones in the audience. Smartphones pervade our everyday life, yet the conventions in theatre-performances usually command them to be turned off while a show is on. Our artists wanted to challenge this social axiom and investigate two main aspects: a) to what extent can a smartphone work as a reception-media in a dance-performance, and b) how can a performance be sensibly augmented through smartphone apps? The artists’ point of departure was to question: how and why did the smartphone change our daily life and relationships. Thus, they wanted to embrace mobile technology for their theatre performance, rather than exclude it, and use it as a way to interact with the audience. RELATED WORK
This work hinges off a line of previous works in different domains, but also from the artists’ own tradition. Due to their interactivity, computer games are one natural ancestor, especially pervasive games of the last decade [1] that tried to blur the line between fiction and reality by breaking out of traditional settings for computer games on screens or theatre performances on stages. There is a line of work on facilitating interactive audience participation in theatre or cinema settings [2]. Early examples of this are Činčera’s Kinoautomat from the Expo 1967 [3] and Carpenters’ SigGraph 1991 Pong Experiment [4] which both brought familiar interactivity into an unfamiliar domain: Kinoautomat used a moderated audience vote to decide how a movie continued; Carpenters’ Pong was an adoption of the early video game for a cinema crowd. Television is another major implicator which devised ways for remote audience votes that could be interpreted and sent back to the public within the frame of the same broadcast. Early examples of these include the so-called “Lichttest” from the 1969 Eurovision show “Wünsch Dir was” and Tele-Dialog (TED, 1979) which was popularized by the Eurovision show “Wetten, dass …?” [5]. The former system measured differences in public electricity-usage to cast audience votes, while the latter system counted telephone-calls to a certain number over a period of time. With the advent of Twitter and Facebook, social media became increasingly popular for gathering audience feedback, both on live-
events like conferences, and also from remote, i.e. for social TV applications. Informed by body-worn technology, generative computer graphics and VJing, there are stage shows that link the kinetic movements of the performers to projected visualizations [6], or vice versa [7, p. 149]. Finally, there is a substantial line of research on electronic response systems in lecture settings [8]. Similar mixed reality performances started out from theatre settings [9] and sought to bring the theatre to the city streets and the wider public [7]. Our work is now concerned with bringing pervasive technology back to the theatre. Prior to this work, the Fabien Prioville Dance Company had embraced videogames and online video-chat in their performances “Jailbreak Mind” and “Experiment on Chatting Bodies”. Smartphones have turned out to be some of the most important technical devices available to today’s consumer and allow people to link with each other whenever and wherever they want. With the presented project, the artists wanted to research the use of smartphones during a dance performance, where people usually have to turn off their phones once they enter the theatre. By using a dedicated app on their phones, the audience can now communicate with the dancers during the show and thus become part of the performance. EXPERIENCE DESCRIPTION
The Smartphone Project is an hour-long piece that is performed in a theatre-setting by two dancers and one actress. It premiered on November 13th & 14th 2013 as part of the Pina40 festival in tanzhaus nrw in Düsseldorf, Germany. Both showings had about 80 visitors each. In addition to the theatre-room, the piece requires one additional room which is used for setup and also during a break-out session for a sub-set of the visitors.
Figure 1. Setup-room and welcome address. For each performance, audience members arrive at the theatre, having bought a ticket in advance, or at the evening cashier. About an hour before the main show starts, they can go to the setup room (see fig. 1. left), where an actress helps install the app on their phone and join the dedicated Wi-Fi network, should they so require. When they join the main room to find their seats, the dancers are already on stage, doing some pre-performance reminiscent of wellknown scenes from Pina Bausch1, e.g. dropping a dancer to
1
Pina Bausch was an influential German modern dance performer, choreographer, and ballet director. She founded
the floor as in the pair-scene of the piece Café Müller (see fig.1, right). Each seat is numbered and audience-members are asked to enter this number into the app. After everyone is seated, the show starts with a welcome address by the same actress who helped people in the setuproom. In a stewardess-like manner she explains the rules for the evening which are different than what is usually expected in the theatre space, and she clarifies potential concerns. For example, she says that phones can be left on and that it is okay to take calls, pictures, video, and share screens with neighbors. Also, the app will not access private data on phones.
Figure 2. First scene, phone rings for someone to pick up. The main-show is structured into about a dozen different scenes that use the phone in different ways. Not all scenes are directly linked with the app, but use the phone in a wider context (e.g., taking videos, flashing in the dark, or separating the users by their phone brand). The following is a brief account of some of the key scenes, each of which uses the phone as an enabling material in unique way. The show starts with a dancer laying on the stage with a phone in his hand (see fig. 2). The stage is set into fog and animated fog also appears on the app. The dancer’s phone starts ringing and some people receive a message, via the app, reading: pick up the phone. As nothing else is happening, people are starting to mumble unless someone has the courage to leave her seat and take the call from the artist’s phone. The actress is on the other end of the line, speaking to the audience member and giving her instructions that include moving chairs from the side of the stage to marked positions on its centre. Upon completion, the dancers are joining the stage and the next scene commences. Linking between scenes is mostly facilitated by the actress who appears as a friendly host, moderator, and instructor on a meta-level, but she is also an integral part of the show that interacts with the dancers. In another scene, as one of the dancers is performing in a seemingly self-satisfied manner to the audience, the actress is standing quietly behind him, typing into her phone (see fig. 3, right). She is live-typing to the audience members’ phones via the app, mocking his dancing. The dancer is not impressed and pushes her off the stage, making space for his egocentric “smartphone-selfie-dance” that he asks the audience to film on video. Many people follow his the Tanztheater Wuppertal Pina Bausch in 1974 which she led up to her death in 2009.
invitation, changing to the camera-app of their phones. The dance itself is tightly timed to the beat of the music and the flashing stage lights. At the end of the scene, it is ensured that people get back to the TSP-app by having the actress parading across the stage, holding up a sign reading: get back to the app.
decide between options A,B, and C via the buttons on the app (see fig. 4). After each decision, the actress is directly putting on the top-voted accessory and the whole scene lasts only for about a minute.
Figure 3. Audience on stage for scanning an AR-scene (left),
Figure 4. Quick live-voting outfit accessories via phone (left), facilitated by pressing moderator (middle).
actress using live-typing as a second screen (right). The divide of the currently dominating mobile operating systems is juxtaposed as the piece physically separates their users into different rooms. At this point, only the iPhone users receive a message, asking them to stand up and leave the room. This act is supported by the actress asking them to go to the “backstage” door, from where they are routed to a separate room. What might seem and feel like a discrimination of the other users at first glance, is culminated when the iPhone users have left the room: lounge music starts playing and the host mockingly reveals to everybody else in a very relaxed manner how happy she is that they are finally gone. The scenes that are then presented to the now separated iPhone and Android users are quite similar and only vary in decoration: people are frozen in time in a certain stage setting and have different markers attached to their bodies. Using their phones, the audience members can approach the performers and inspect the scene. At this point, the app utilizes standard marker tracking to provide a “magic window” view that reveals certain clues about the scene on the smartphone’s screen as the markers are being recognized. The scene depicted in figure 3 (left) reveals additional information about the actress, while the users in the other room are presented with a crime-scene, where evidence and the weapon are only visible in augmented reality (AR). Extending on the use of music and loudspeakers in dancetheatre, the music-scene provides a transition of music from the theatre speakers to the smartphone-speakers as the artists leave the stage. This element was particularly challenging to implement. Keeping the music synchronous to the beat across a range of different phone models and two operating systems meant dealing with audio-latencies from below 10 ms (iOS) up to over 350 ms (some Android). The last component presented in this paper is the livevoting via the app. The objective is for the audience to quickly and mutually agree on a dress for the actress (e.g. wig, dress, and belt). This is facilitated by one of the dancers taking the role of a moderator who is advertising the different options to the audience, urging them to quickly
RESULTS
To support our study, we devised a survey with 22 open and closed questions, asking questions about the performance in general and particular elements thereof; trying to identify the strongest and weakest interactions; how the technology could be further enhanced or used for other events; and if participants would recommend or revisit a piece like this. From the roughly 160 premiere guests on both evenings, we received 36 filled surveys (about 20%). Gender distribution was 64% female and 36% male. Operating system distribution was 39% Android and 42% iOS. Twenty-eight percent of the users installed the app before coming to the theatre. Sixty-one percent of the users shared their phone with their neighbors during the experience. We asked a range of questions with a rating scale ranging from 1 (Very Bad) to 5 (Very Good). The performance scored an average of 4.48 (SD=0.74). The general idea of using smartphones for audience interaction in theatres scored 4.29 (SD=1.07). The actual implementation of this idea in the presented piece scored 4.09 (SD=0.97). Of the specific interactive elements of the show, “live-typing” (4.49, SD=0.89) and “voting” (4.72, SD=0.84) were rated highest. The augmented reality scene scored 4.30 (SD=0.91) and the music transition scored 4.47 (SD=0.86). Summing up the closed questions, 3% knew similar work (the one who did, knew another piece by the same company), 84% stated that the performance was enriched by using smartphones, 81% would come again, and 83% would recommend the performance. The open questions allowed for criticism and suggestions for improvements. Some people were unsatisfied by their own expectations and wanted more AR at the “transition” scene; while others suggested to also have smartphone-free elements and more dancing. It was suggested to provide better dramaturgic linking between the elements of the show, and to use the smartphone for more interaction, like being able to respond to live-typing. Generally, the “sharing is caring” approach was seen as a neat opportunity to get in contact with the neighbor. Applications for similar systems
were seen in quizzes, other types of theatres, stand-up comedy shows and also interacting with light installations. LESSONS LEARNED
Our work departed from the artist’s idea to build a piece on the topic of smartphones and augmented reality by using those technologies as a medium (“the piece is the app”). Arriving at the presented system is a process that might be subject to a paper on its own. But even at the early phases, it was already clear that the piece had to manage expectations and concerns on two levels: the relation of the piece to the work of Pina Bausch’s, and the protection of people’s personal phone data, e.g. not tossing their private pictures into the public. This was addressed in the welcome scene, which explained the rules of the show and then directly led to the first scene (see fig. 2). From our experience, we found the setup help an essential ingredient that must be considered when building similar work. Augmented Reality was relatively unknown to our audience: 78% had not used any other AR application. Many participants liked AR, and they thought it was simple to use and enhanced the performance (i.e., it made them leave their seat and explore). Several participants stated that they would like to see more AR in future performances, and one participant stated that “more markers more own exploration”. It also seemed like a good excuse to get people off their seat. The music element, where music from the big speakers was transferred to the audience’s phones when the artists left the stage, received mixed results. Many participants thought the music was off beat, as they assumed the music was intended to be synchronous. While augmented reality and the synchronized music were the most technically challenging elements, it is interesting to observe that the most popular elements of the show were live-typing and voting. Our interpretation of this is that while bespoke technology makes us curious, we also still like to follow interesting stories and get involved. In the live-typing scene, the artists made use of this simple mechanism to build up tension between what was seen on the stage and what people might say about it when asked in private. This tension between public performance and private screen is thus an interesting seam for creating meaningful interactions. Another such seam is the prevalent discussion of operating systems (Android vs iOS) and their underlying beliefs. This was picked up by actually physically separating the users of those groups for the AR scene into separate rooms, and then merging them back for the next scene with a harmonic dramaturgy. iOS users made a procession with their screens lit as they entered the dark theatre to fitting music, while Android users were constantly flashing the darkness by taking random pictures. In essence, the piece provided a complex frame for novel, but ultimately simple interactions under an artistic umbrella. By introducing the smartphone into the theatre space and continuously questioning the rules of that space,
the piece provided the ground for some quite simple lowtech interactions like asking audience members for music from their phones and hooking it up to the theatre’s speaker, and then having the dancers improvise on that music. We found that not everything had to be implemented in the app, e.g. as it is easier and more charming for the performer to ask audience members to take videos or go back to the app, than to implement all cases in software. Moreover, the liberal approach to letting the audience take pictures and videos of the performances and share them online basically aided the artists’ marketing for free. ACKNOWLEDGMENTS
Produced in collaboration with tanzhaus nrw and the Ballet National de Marseille with funding from Ministerium für Familie, Kinder, Jugend, Kultur und Sport des Landes NRW, Ministerium für Innovation, Wissenschaft und Forschung des Landes NRW, Kunststiftung NRW, Kulturämter der Landeshauptstadt Düsseldorf und Stadt Wuppertal, as well as Stiftung van Meeteren. Images courtesy of Mischa Lorenz and mehtafilm. REFERENCES
[1] S. Benford, C. Magerkurth, and P. Ljungstrand, “Bridging the Physical and Digital in Pervasive Gaming,” Commun. ACM, vol. 48, no. 3, pp. 54–57, 2005. [2] D. Maynes-Aminzade, R. Pausch, and S. Seitz, “Techniques for interactive audience participation,” in Fourth IEEE International Conference on Multimodal Interfaces, 2002. Proceedings, 2002, pp. 15–20. [3] “Kinoautomat.” [Online]. Available: http://www.kinoautomat.cz/. [Accessed: 03-Jan-2015]. [4] “Cinematrix Inc.” [Online]. Available: http://usa.cinematrix.info/. [Accessed: 03-Jan-2015]. [5] M. Denninger, “TED oder die Tele-Demokratie.” [Online]. Available: http://www.zdfjahrbuch.de/2004/produktion/denninger.htm. [Accessed: 03-Jan-2015]. [6] B. Gonzalez, E. Carroll, and C. Latulipe, “Danceinspired Technology, Technology-inspired Dance,” in Proceedings of the 7th Nordic Conference on HumanComputer Interaction, NY, USA, 2012, pp. 398–407. [7] S. Reeves, Designing Interfaces in Public Settings (Human-computer Interaction Series): Understanding the Role of the Spectator in Human-Computer Interaction, 2011th ed. Springer London Ltd, 2013. [8] E. Judson and D. Sawada, “Learning from Past and Present: Electronic Response Systems in College Lecture Halls,” J. Comput. Math. Sci. Teach., vol. 21, no. 2, pp. 167–181, 2002. [9] S. Benford, A. Crabtree, S. Reeves, J. Sheridan, A. Dix, M. Flintham, and A. Drozd, “The Frame of the Game: Blurring the Boundary Between Fiction and Reality in Mobile Experiences,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2006, pp. 427–436.
