Development of a Protocol for Tangible User Interfaces Evaluation: The Case of Brazilian Motorcycle Simulators
ABSTRACT
Tangible user interfaces (TUIs) allow the users to experience full embodiment in order to promote interactions between physical objects/actions and virtual scenarios. This article presents the development of a protocol for TUI evaluation from the usability analysis of a motorcycle simulator. The objective of this research, funded by the Brazilian Federal Government, was previously to establish requirements for an appropriate TUI that meets the reality of Brazilian users. Further research involved ethnographic procedures (observations and interviews with several social actors in driving schools) and usability tests (performed in a usability laboratory with 20 driving schools students). As a result, a protocol for evaluation of products with TUI has been proposed. Six dimensions of analysis were considered in such protocol: environment, user, instructor, physical object, virtual scenery, and physical-virtual interaction.
Author Keywords Tangible user interfaces; Usability; Evaluation. ACM Classification Keywords H.5.2. User Interfaces: Theory and methods. INTRODUCTION
Tangible User Interfaces (TUI) emerged as an object of study in Design about 18 years ago, following the classic article by MIT’s researcher Hiroshi Ishii [9]. This type of interface comes from technologies like Augmented Reality and Virtual Reality [17], and differs from the Graphical Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected] or Publications Dept., ACM, Inc., fax +1 (212) 869-0481.
User Interfaces (based on audiovisual stimuli and operating primarily through screen, keyboard and mouse). TUIs go beyond the WIMP visual paradigm (acronym for Windows, Icons, Menus and Prompts) [11], and are related to the manipulation of physical objects [18], or the elicitation of physical gestures or actions, to promote interactions with actions in virtual scenarios [9, 16]. The design of Graphical User Interfaces is assisted by protocols and guidelines for several types of evaluation [14]. Yet, these kinds of evaluation procedures are rare for TUIs: “unlike traditional Human-Computer Interaction, there has been relatively little effort to establish how TUIs can be designed from scratch” [23]. The aim of designing and evaluating TUIs is a challenge for designers of interfaces, given that such products are poorly understood in terms of requirements and constraints. Due to increasing technological and commercial feasibility, this type of interface is becoming more common. Thus, with the spread of Augmented Reality and other interaction resources, designers will need instruments to deal with TUIs [5]. The research presented in this article aims to develop a protocol for tangible user interface evaluation. The term protocol in this sense means a structured and systematic process for widespread and flexible use, with a modular nature [13]. In other words, a set of instructions on how to evaluate a tangible interface product, which can be adapted for several types of these products. To accomplish this goal, a set of usability tests of a motorcycle simulator was conducted, making use of TUIs. The procedure was complemented with ethnographic observations and interviews with users in natural settings (driving schools). The research presented in this paper is part of a Brazilian Federal Government initiative to establish interaction requirements for driving schools simulators throughout the Brazilian territory. As a result, a protocol for evaluation of tangible user interfaces was developed. The protocol has six modules (defined based on literature review), each one with a set of
specific items (coming from the ethnographic research and usability tests). The protocol developed demands more tests for widespread validation, as it does not include all types of tangible user interfaces. Instead, it focused on products used for classroom education, especially through simulation training. SIMULATORS AS TANGIBLE USER INTERFACES Interface and Human Cognition
TUIs may be defined as interactions that make use of two basic elements: a physical product (which is manipulated by one or more users) and a virtual environment (which receives commands through the physical product). Thus, “Tangible Bits allows users to grasp and manipulate bits in the center of users’ attention by coupling the bits with everyday physical objects and architectural surfaces” [16]. In physical-virtual interaction, certain factors stand out, such as: a) The user’s physical responses, such as movements in the environment [23] and the manipulation of objects [18]; b) Naturalness of the interaction [24], i.e., the use of well-known motions, thus enabling an intuitive interaction. Naturalness is possible because the TUIs are based on metaphors of behaviors known by the users. TUIs: “are designed to recall in their physical appearance the properties and behaviors of real world objects familiar to the user, so that an intuitive use of them should be sufficient to exploit the functionalities of the computing system” [10]. On the use of metaphors to project natural user interfaces: “We believe that all of these new interaction styles draw strength by building on users’ pre‐existing knowledge of the everyday, non‐digital world to a much greater extent than before. They employ themes of reality […]. They thereby attempt to make computer interaction more like interacting with the real, non-digital world” [8]. Because of the use of metaphors and physical objects, TUIs are referred to by Jetter [11] as “Reality-Based Interaction”. Jetter established four classes of criteria for the evaluation of the use of tangible interfaces, which he called “Four Domains of Blended Interaction”: 1) Relation between the user and the system; 2) Social relations between users and other individuals without the system; 3) Collaboration of individuals within the system; 4) Physical setting where all the interactions occur. Jetter [11] highlights the role of physical environment and social experience of the individuals in the interaction, which brings out the meaning of embodied cognition. The human cognition explored on a TUI is substantially embodied: as explained through Phenomenology and Ecological Psychology, based on “sensorimotor coupling in a social situation” [4]. From an embodiment perspective, physical and virtual are not two distinct dimensions that interacts mechanically, but
two aspects of being-in-the-world. For this reason, according to [8] the design of a TUI must be oriented by some principles of human cognition, such as: Social situatedness (focus on interpersonal interactions in real-life environments); The use of “Scaffolds” (tools that enable people to easily perform tasks); The presence of commonly known meanings present in the environment, like instructions and tips; The use of interactive imagery in hands-on activities (instead of executing mental plans); Dialogical systems in which people are engaged in realtime communication with each other and, at the same time, mediated by information systems; etc. On the other hand, by focusing on the matching of physical and virtual events, [22] elucidates six aspects of these relations: Time (Continuity between user action and system response); Location (Space correspondence between user action and system reaction); Direction (Direction matching. Ex: Left, Right, Up, Down); Dynamicity (Position, velocity, acceleration, force); Modality (The system provides reactions on the same sensory mode of user actions); Expression (When the system reaction reflects the emotional expressions of user’s action). Motorcycle simulators
According to [6], a vehicle simulator is any device able to reproduce the conditions of using a vehicle in a virtual scenario, which includes an environmental interaction. Vehicular simulators can be addressed as industrial applications with TUI since such products usually make use of physical objects (chassis of a motorcycle, for instance) to promote interaction with virtual scenarios. A simulator is effective in its educational goals if the learning is done in a way which is transferable to real-life situations. The main reason given as a determinant for this learning transfer is the sensorimotor realism of the simulation [19]. Such realism demands the alignment of two elements: user sensory-motor commands (obtained by physical parts such as handlebars, gears, buttons, etc); and the consequences of these commands in the virtual environment. Driving simulators are used for training enactive abilities in its users, i.e. cognitive skills that require sensorial and motor activities [15]. Hence, the presence of real movement in the simulator (kinesthesia) consists in a fundamental question to design such products [12]. Simulators with such feature provide better driving enactive abilities learning in its users, when compared to the ones without movement[1].
However, the presence of kinesthesia raises the price of simulators in terms of manufacturing and maintenance costs [6].
This article is based on a research conducted by the Brazilian federal government between the years 2013 and 2014 on the evaluation of TUI in motorcycle simulators.
Besides the kinesthesia issue, according to [20], the challenge of designing a two wheeled vehicle simulator is to promote a balance between the reality of the simulation (which increases the financial costs), with the expertise level of the target audience, in order to guarantee the object viability for consumption on a large scale.
METHODS
Therefore, to novice drivers, and students with no experience, a low level of sensory and motor ability is required, resulting in a simulator of lower cost [2]. However, the definition of an appropriate balance between realism, cost and efficiency for novice students poses a difficult design topic. Because of this design issue, the use of motorcycle simulators worldwide, compared to cars, is reduced [12].
The interviews were conducted with driving instructors, students and driving schools owners, focusing on users’ constraints and needs. Participant-observations of the motorcycle simulators used in training centers in three Brazilian states were conducted. The aim of the first phase was to involve different types of users in the design process, thereby promoting a User Centered Design of a prototype.
The research presented in this paper focused on motorcycle simulators for driving school designed for the purpose of awareness training and risk control in traffic for inexperienced students. As an example of this type of simulator, there is the Honda Riding Trainer, which "improves the novice riders' ability of recognizing hazard situations and reacting in such a way to avoid the risks" [12]. Thus, the design of motorcycle simulators demands the understanding of driver’s sensorimotor behaviors, and the physical conditions of traffic. To establish requirements for TUIs, driving simulators demand the mapping of user’s actions in a given driving context, the establishment of simulator’s physical attributes, and relations between user’s physical responses and virtual effects in the simulation. Brazilian research requirements
for
simulators’
The research to deploy the TUI evaluation protocol was structured in two phases. The first phase consisted of ethnographic procedures: participant-observations and onsite interviews. This phase lasted three months and was carried out in driving schools.
The second part of the research consisted of prototype usability tests. The tests involved 20 volunteers, all students of driving schools pursuing a motorcycle driving license. The sample consisted of 10 men and 10 women, between the ages of 18 and 45. From the 20 participants, 16 were enrolled in high school, and 4 in university; 7 participants had no experience with motorcycle driving, 3 already were able to drive a motorcycle, and 10 had some experience. A representation of the simulator prototype is shown in Figure 1:
interfaces
Brazil has 42,000 deaths per year in traffic, with a 3.7% increase annual trend [21]. The problem is aggravated in motorcyclists, since between 2000 and 2011 the fleet increased from 4 to 18 million, and the accident rate increased 932.1%; the deaths of motorcyclists are almost 14,000 per year in Brazilian traffic. Several factors explain this alarming rate: infrastructure problems on roads, lack of quality public transportation, and, as one of the main determinants, lack of driver's awareness [7]. In order to increase the risk management training of novice motorcyclists in driving schools, the Brazilian Federal Government has been developing studies on the use of educational simulators. In 2010, a series of studies had been promoted to determine the basic requirements of most suitable car simulators for Brazilian driving schools. These studies substantiate a federal law that regulates the design of such products (the Federal Ordinance 808, from 2011). In 2013, a similar study was conducted for motorcycle simulators.
Figure 1 – Visual prototype scheme.
Figure 1 represents an instructor operating the simulator controls (1), and the motorcycle product (2) without any user. The usability test consisted of systematic observations conducted in special control conditions: the testing environment was the LabUP, a usability lab at the Sapiens Park Institute in Florianopolis. In Figure 2, a student is
shown while using the motorcycle simulator developed for the tests:
Figure 2 - Student using the simulator.
the actions in a virtual scenario were established as the main characteristic of TUIs. In other words, there were three dimensions of analysis: physical product, virtual scenario and relations between them. The environmental importance, established by [11], was the physical and organizational space inside the driving schools, and the social relations between different types of users. From the ethnographic and usability test, two types of users were established: the student and the instructor. Thus, three more dimensions of analysis: environment, user-student and user-instructor. The protocol developed in this research was called "Tangible User Interfaces Assessment Protocol for Simulators" (TUIAPS). Figure 3 presents the TUIAPS’ six dimensions of analysis:
The tests were focused on observing students while they were performing tasks and learning basic driving skills (which includes triggering the bike, changing gears, following safely on track, stopping the bike, etc). In addition to direct observation of the use of the simulator, participants were also interviewed regarding satisfaction with the simulator. All the ethical precautions for testing the simulation with people were taken into consideration. The tests were approved by UDESC Ethics Committee, in Florianopolis (Process code: CAAE 32278414.7.0000.0118). RESULTS Main Ethnographic and Testing Findings
The importance of carrying out a motorcycle simulator design taking into account the social and organizational needs of driving schools was reinforced by the ethnographic research. For instance, to conceive the whole flux of tasks involving classes in the simulator, and how it defines issues on the physical product such as: location, room space dedicated for it, physical configuration of the furniture in the room, etc. Moreover, the observations and interviews also showed the major concerns of instructors in operational terms, such as ways of controlling the interface and the need to learn about the simulator’s technology not related to their professions. The usability test also demonstrates how the simulator’s TUI is based on sensory realism (visual, haptic, auditory and kinesthetic) and accurately pairing user’s driving responses and the simulation’s consequences [19] in the six relations mentioned by [22]. All 20 participants declared to be highly satisfied with the experience; 15 of those assured that the simulators’ kinesthesia provided an increase of selfconfidence when practicing with a real motorcycle. Nonetheless, 9 mentioned a reduced, yet noticeable lack of realism with the sound and synchronization between their actions and what happened in the simulation. The developed evaluation protocol
From [9, 23], the relations between a physical product and
Figure 3 - The six modules of TUIAPS.
In Figure 3, the six dimensions of analysis are listed in accordance with a suggested sequence of evaluation, ranging from most broad to most specific: Environment (1) User-Instructor (2), User-Student (3) Tangible Product (4), Intangible Scenario assets (5), and Tangible-Intangible relation (6). These dimensions are related as follows: (2), (3) and (4) are inserted in (1). The set (1) (2) (3) (4) is integrated with (5) through (6). Following this suggested ordinal sequence, the evaluation process begins with the environment where the interaction takes place. When it comes to the physical environment, it highlights the importance of a comfortable space (to the product, one instructor and one or more students). About other physical aspects of this space, the thermal comfort proved to be an important aspect (taking into account using a helmet and the hot climates in Brazil). The proper lighting also appears in this list, to help the users staying focused. The minimum needed is an environment without external noises. The use of time also appears as a major question. A better solution to create a special agenda for simulation training was defined, instead of integrating these classes with regular ones in the same period. The research showed that, because the motorcycle simulator is a product to be used in an educational institution, the organizational aspect of the environment should also be considered. The main items are: flow of students in the
class; sound-proofed setting to prevent the simulator from being a source of noise in driving school; simulator visibility to promote the product in question. The second dimension of analysis is the User-Instructor, which is a special type of user that makes the mediation among the student and the simulator. The User-Instructor has an institutional role guaranteed by law within a driving school [20]. The User-Instructor must have experience in both conducting the simulated vehicle, and operating the simulator. There is the possibility of fatigue in the case of many sessions on the simulator, therefore the UserInstructor requires rest for every two hours of work. Additionally, the instructor must have opportunities and ways to properly communicate with the student during the session [6]. The third dimension of analysis is the User-Student. The minimum requirement for the simulator’s use is that the individual does not present symptoms of dizziness and malaise resulting from experience, common in a small part of the population and known as "Simulation Sickness”[2]. In the usability test performed, one case of this undesirable effect was reported (the user could not continue the testing sessions). Thus, the simulator should provide conditions for use taking into account the sensorimotor capabilities and constraints of the User-Student. In other words, it is important that it be accessible to users of different heights, weights and body proportions. The User-Student must be informed that the TUI present in the simulator works as a metaphor [3] for a real motorcycle. Thus, it does not recreate a perfect interaction like in the real vehicle: the simulator only reproduces part of this interaction to promote training on its enactive skills [15, 19], such as driving hazard management. The level of the User-Student’s expertise in vehicle driving did not affect the use of the TUI, which is the same for all users. Tests with users with special needs were not conducted, such as physical or sensory disabilities. The fourth dimension of analysis is the Tangible Product. It was determined that it must be very realistic in terms of sensorimotor experience for the users, in order to promote the naturalness of the interaction [24]. In other words, the cabin where the user-student stays should look like the motorcycle being simulated: seat, panel, position, physical properties of the commands, etc. The reliability of Tangible Product toward the simulated vehicle helps in the user's’ immersion. A realistic physical cabin provided a rich model of real motorcycles to users [1]. The set of those four dimensions establishes what [8] call as “social situatedness”, and includes other factors, such as use of “Scaffoldings” and shared traces of meanings present in the environment. The research has demonstrated that this social and embodied environment must be evaluated before the other dimensions. This is because the sensory aspect of the experience emerges, in a major part, from the four aforementioned dimensions.
The fifth dimension of analysis, Intangible Scenario, involves planning in educational terms to ensure that it accomplishes instructional goals. Virtual scenarios must comply with local features and the traffic laws of the country where the simulator is used [1], and to be evaluated with specialized pedagogical support when the evaluation team lacks the pedagogical knowledge. The sixth dimension of analysis, Tangible-Intangible Relation, should be synchronized in various ways to ensure naturalness of use, i.e. a truly intuitive and rich feedback for the User-Student’s physical responses. The main relations were mentioned by [22]: Time, location, Direction, Dynamicity, Modality and Expression. The usability test has demonstrated that the most important ones, for driving simulators, are Time, Location, Direction, and Dynamicity. A lack of time synchronization (sometimes less than one second) is the most common problem, and it decreases the feedback of User-Student’s responses. CONCLUSION
The design of interfaces needs to adapt to new ways of interaction provided by technological innovations. Developing standards, patterns and evaluation protocols for TUIs are important parts of this technical and methodological challenge. Interaction designers must become accustomed to TUIs, given that this type of interface will become more common due to dissemination of resources such as Augmented Reality. In order to prepare these professionals, evaluation protocols, as well as other instruments, are necessary. Beyond this instrumental innovation, the study of new research methodologies is also required, especially ethnographic procedures. These procedures have shown how it is necessary to consider different user types acting together, in a specific social environment, which has physical properties and organizational rules. Designing TUI demands, therefore, an understanding of human cognition, which requires theoretical reflection on approaches like Embodied Cognition, Phenomenology and Ecological Psychology. The TUIAPS, in its current version, can serve as a tool of TUI requisite evaluation for interaction designers. It can be used as a checklist of requisites, especially in TUIs with educational purposes in the classrooms. The TUIAPS does not solve all cases of TUIs. This protocol focuses on simulators and other products for educational purposes. As it lacks quantitative validation, its use demands flexibility and adjustments for specific cases. Suggestions for future studies include: a) Implications of Embodied Cognition for design and evaluation of TUIs ; b) Training Programs for physical skills through TUIs; c) Design of TUIs for simulators focused on physical performance, such as: dance, martial arts, musical instruments, etc.
ACKNOWLEDGMENTS
We want to thank our friends Fred van Amstel, Pedro Saynovich, and Lui Pillmann for their support and feedback on this article. REFERENCES
1.
Wade Allen, George Park, Marcia Cook, Dary Fiorentino. 2007. The Effect of Driving Simulator Fidelity on Training Effectiveness. In Proceedings of the DSC 2007, 323-345. 2. Timothy Baldwin, Jason Ford. 2006. Transfer of Training: a review and direction for the future research. Personnel Psychology 41, 2: 121-142. 3. Augusto Celentano, Emmanuel Dubois. 2014. Metaphors, analogies, symbols: in search of naturalness intangible user interfaces. In Proceedings of 6th International Conference on Intelligent Human Computer Interaction (IHCI’2014), 100-114. 4. Jelle van Dijk, R. Mitchell. 2014. Co-Embodied Technology: A Design space for Human Being. In Proceeding of Tangible and Embedded Interactio (TEI’14) WIP poster. 5. Maria Goulart Braga. 2012. Diretrizes para o design de mídias em realidade aumentada: situar a aprendizagem colaborativa on line. Ph.D Dissertation. UFSC, Florianopolis, Brazil. 6. Peter Hancock, Dennis Vincenzi, John Wise, Mustapha Mouloua. 2009. Human Factors in Simulation Training. CRC Press 7. Maria Hoffmann, Roberto Cruz, Marcos Alchieri. 2003. Comportamento humano no trânsito. Casa do Psicólogo. 8. Caroline Hummles, Jelle van Dijk. 2015. Seven Principles to Design for Embodied Sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '2015), 2-13. 9. Hiroshi Ishii, Brygg Ullmer. 1997. Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’ 1997), 1 - 8. http://web.media.mit.edu/~anjchang/ti01/ishiichi97-tangbits.pdf 10. Robert Jacob, Audrey Girouard, Leanne Hirshfield, Michael Horn, Orit Shaer, Erin Treacy Solovey, Jamie Zigelbaum. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In Proceeding in Computing Systems (CHI ’08), 201-10 11. Hans-Christian Jetter. 2013. Blended Interaction: Envisioning Future Collaborative Interactive Spaces. In Proceedings of the CHI Conference on Human Factors in Computing (CHI’2013). 12. Lamri Nehaoua, Salim Hima, Hichem Arioui, Nicolas Seguy. 2009. A New Motorcycle Simulator Platform: Mechatronics Design, Dynamics Modeling and Control. In Proceeding of 17th World Congress, International Federation of Automatic Control
(IFAC’08), 1-14. 13. Jakob Nielsen. 1993. Usability Engineering. Morgan Kauffmann. 14. Jennifer Preece, Yvone Rogers, Helen Sharp. 2005. Interaction Design: Beyond Human-ComputerInteraction. Bookman Press. 15. John Protevi. 2007. Beyond Autopoiesis: Influences of Emergence and Politics in the work of Francisco Varela. Retrieved July 23, 2015 from http://www.protevi.com/john/Varela.pdf 16. Orit Shaer, Evan Hornecker. 2010. Tangible User Interfaces: Past, Present, and Future Directions. Foundations and Trends in Human–Computer Interaction 3, 2: 1–137. 17. Romero Toro. 2010. A presença das tecnologias interativas na educação. Revista de Computação e Tecnologia da PUC-SP 2, 3: 3-16. 18. Bryggs Ullmer. 2002. Tangible User Interface for Manipulation Agregate of Digital Information. Ph.D Dissertation. Massachusetts Institute of Technology, Cambridge, MA. 19. Giulio Vidotto, 2011. A Longitudinal Study to Evaluate riding Trainer Effectiveness in Teenagers. In Proceeding of 3rd International Conference on Road Safety and Simulation, 14-22. 20. Ricardo Vieira, Manuel Steidle, Alessandro Vieira, Ricardo Pralon. 2014. Conceptual Architecture of Motorcycle Simulators for the Training of Novice Riders. In. Proceeding of International Motorcycle Conference (IFZ’2014), 34-52. 21. Julio Waisefisz. 2014. Mapa da Violência – Acidentes de Trânsito e Motocicleta. Retrieved April 20, 2014 fromhttp://www.mapadaviolencia.org.br/pdf2013/ma pa2013_transito.pdf 22. S. A. G. Wensveen, J.P. Djajadiningrat C.J. Overbeeke. 2004. Interaction Frogger: a Design Framework to Couple Action and Function through Feedback and Feedforward. In Proceedings of Designing Interactive Systems Conference (DIS’04), 1-12. 23. Alexander Wiethoff, Hanna Schneider, Julia Küfner,Michael Rohs, Andreas Butz and Saul Greenberg. 2013. Paperbox – A toolkit for exploring tangible interaction on interactive surfaces. Department of Computer Science, University of Calgary, Calgary, AB, Canada. 24. Bieke Zaman, Vero Abeele, Panos Markopoulos, Paul Marshall. 2011. Editorial: the evolving field of tangible interaction for children: the challenge of empirical validation. Personal and Ubiquitous Computing 16, 4: 367-378.
