Human Adaptation to a Miniature Robot - Semantic Scholar

2 downloads 0 Views 184KB Size Report
utilize these and related findings for building robots not only capable ... Building robots that can exhibit such kind .... 2: The e-puck Robot used in the experiment.
Human Adaptation to a Miniature Robot: Precursors of Mutual Adaptation Yasser Mohammad Graduate School of Informatics Kyoto University Yoshida-Honmachi, Sakyo-ku, Kyoto 606-8501 Japan

Toyoaki Nishida Graduate School of Informatics Kyoto University Yoshida-Honmachi, Sakyo-ku, Kyoto 606-8501 Japan

[email protected]

[email protected]

Abstract— Mutual adaptation is an important phenomenon in human-human communications. Traditionally HRI research was more interested in investigating adaptation of the robot to the human using machine learning techniques but the possibility of utilizing the natural ability of humans to adapt to other humans and artifacts including robots is recently becoming increasingly attractive. This paper presents some of the results from an experiment conducted to investigate the interaction patterns and effectiveness of motion cues as a feedback modality between a human operator and a miniature robot in a confined collaborative navigation task. The results presented in this paper show evidence of human adaptation to the robot and moreover suggest that the adaptation rate is not constant or continuous in time but is discontinuous and nonlinear. The results also show evidence of a starting exploration stage before the adaptation with duration dependent on the expectations of the human regarding the capabilities of the robot in the given task. The paper investigates how to utilize these and related findings for building robots not only capable of adapting to human operators but can also help those operators adapt to them.

I. I NTRODUCTION Currently robots are starting to move to our houses and offices. Many of those robots do not have a humanoid body because of the specific tasks they are designed to carry on. This fact makes it important to study the interaction between humans and non-humanoid robots. Interaction with nonhumanoid robots poses special challenges that do not exist in the humanoid case including reduced anthropomorphism and the lack of predefined nonverbal protocols. Interaction with miniature non-humanoid robots poses more challenges including the difference in size, the limited degrees of freedom that are usually available to the robot to express its internal state, and the limited computational resources. Mutual adaptation is a known phenomenon in humanhuman interaction scenarios confirmed by controlled experiments [1]. Building robots that can exhibit such kind of mutual adaptation can improve the acceptability and usability of robots especially in collaborative scenarios. Recently researchers in HRI are becoming more interested in investigating mutual adaptation in human-robot interaction scenarios. In [2] a humanoid robot was developed that

can attain incremental learning and shows signs of mutual adaptation in a navigation task by utilizing recurrent neural networks. The system showed improved learning even in long interaction times. [3] developed an interactive system named HUMA for generating a behavioral protocol to enable physical contact between a human and a robot. HUMA employs a simultaneous method that continuously maintains a protocol database while manifesting the protocol by carrying out robots actions. The simultaneous method attempts to achieve mutual adaptation between a person and the robot. [4] explored the use of a hierarchical action control architecture inspired by the theory of simulation to learn the nonverbal interaction protocol in more general settings (that do not involve physical contact) and reported fast learning of the interaction protocol in a simulation study. To be able to build mutually adaptable robots it is essential to understand the adaptation behavior of the human as well as the robot. Adaptation of the robot to the human commands was studied by many researchers. For example In [5] the physiological signals of the participants were monitored in real time using wearable biofeedback sensors and Biopac data acquisition system and used to infer the affective state of the human which was used to help the robot adapt to its partner using a control architecture inspired by Riley’s information flow model. [6] studied the way humans tend to teach robots using a simulated environment in which a human is teaching a simulated robot (Sophia) how to cook and discovered a tendency to use the feedback channel from the operator to the robot not only for awarding but as a suggestion channel despite the fact that the operator was told that this is not a valid use of it. Based on this results the Q-Learning algorithm was modified to account for this tendency and a significant improvement in the learning speed was reported. This results were obtained in an explicit teaching situation. In a more implicit settings, [7] designed a system to help the robot adapt to environmental noise during interaction with humans by controlling its location and orientation. [8] studied the adaptation of a robot manipulator to fuzzy verbal commands using Probabilistic Neural Networks and reported successful learning with a PA-10 redundant manipulator Adaptation of the human to the robot capabilities is less studied. For example [9] compared using predefined gestures

and a joystick to control a miniature robot in the same environment used in the experiment reported in this paper and found that the gesture based interface was more efficient and reduced the average task completion time compared with the joystick interface. Their experiment also showed a form of human adaptation to the robot capabilities. The observed adaptation behaviors where: 1) skip behavior where the human skipped some commands when (s)he discovered that the robot can do without it 2) replace behavior when the human replaces a gesture that fails to achieve the goal with another corrected one 3) discovery behavior where the human adjusts the way (s)he performs the gesture to the behavior of the robot. These results suggest that humans can adapt to the capabilities of miniature robots in navigation control tasks. In this paper we are interested in investigating the existence and the properties of this adaptation in a collaborative navigation scenario. This paper reports results from an experiment to study gesture interface to a miniature robot concerning human adaptation to the robot. Three main characteristics of human adaptation were found in this experiment. Firstly, adaptation started after an exploration stage with low adaptation and slowed down by the end of the interaction in an adaptation saturation stage. Secondly, the duration of the exploration phase and the rate of adaptation through out the interaction depends on the initial expectations of the human concerning robot capabilities in the task at hand. Lastly, the subjects showed signs of task adaptation earlier than signs of adaptation to the robot capabilities. The paper also discusses how these results can be used to build robots that can not only adapt to humans but can help humans adapt to them by respecting these features of human adaptation. The rest of the paper is organized as follows. The following section briefly introduces the experimental setup followed by the experimental procedure. The results of the experiment concerning analysis of human adaptation are then reports and then the paper is concluded with final remarks about implications of the three main findings from this analysis for interactive robots. II. E XPERIMENTAL S ETUP The experiment is designed as a game in which a human is instructing a miniature robot using free hand gestures to follow a pre-specified path to a goal in an environment that is projected on the ground. The path is visible to the human but not the robot. Along the path there are five virtual objects. Those objects are visible only to the robot through a set of virtual infrared sensors. The only way the human can give the robot the correct instructions is if the robot could transfer its knowledge about those objects or the recommended actions regarding them to her/him. The essential property of this design is that the partially observable environment can be converted into a completely observable environment if and

Fig. 1: A simplified version of the environmental setup. The magic mirror allows the hidden operator to see the main operator while the main operator is unaware of his existence

Fig. 2: The e-puck Robot used in the experiment

only if the human and the robot succeeded in exchanging their knowledge. The experimental setup is shown in Fig. 1. The human subject (called the main operator from here on) stands in front of a 60×40cm rectangular working space in which the robot is free to move. The required path is projected upon this environment so the main operator knows the path the robot should be following. A magic mirror separates the main operator from a hidden human operator (called the hidden operator from here on) whose responsibility is to recognize the gesture the main operator is issuing and to input it to the Gesture Transfer Software which sends it to the Robot Control Software. The Robot Control Software is responsible of controlling the behavior of the robot based on the gestures of the human and the detected objects in the environment (which can not be seen by the main operator). A sealing camera was used to localize the robot during this experiment. The localization algorithm used elliptic markers to accurately find the location and direction of the robot within the working area. The robot used in this experiment is a miniature robot called e-puck designed by EPFL. Figure 2 shows the robot with and without the localization marker attached. The diameter of the robot is 7cm, and is driven by 2 stepper motors in a differential drive arrangement. The hidden human operator is needed in this experiment for two reasons. First the gestures he enters to the Gesture Transfer Software were used to find the gestures that are usually used by the user and to analyze them to find signs of human adaptation. The second reason for using a hidden operator is to control the conditions of the experiment so that

based on an earlier pilot study reported in [10]. III. P ROCEDURE

Fig. 3: The GUI of the software used by the Hidden Gesture Recognizer

the efficiency of achieving the task depends mainly on the understandability of the feedback from the robot rather than the accuracy of the gesture recognizer (The hidden operator is assumed to be a perfect gesture recognizer). Using a virtual projected environment was a must in this experiment as it is the only way to make the main operator see the path while not seeing the objects on it. For the details of the localization system and the robot control software refer to [10]. A. GT software The Hidden Gesture Recognizer uses the GT software shown in Fig. 3 to send the commands recognized from the free gestures of the main operator to the robot. It should be stressed here that the Hidden Gesture Recognizer in the experiment does not control the robot but is utilized by the robot as an external sensor. The software consists of four main areas: 1) The upper left area shows the path with the robot projected over it as seen by the localization system and is used to transfer pointing gestures (by clicking on the pointing target point) 2) The lower left area transfers a real time video image of the environment. 3) The upper right area shows the command window and is used to transfer all the other kinds of gestures understood by the system. This software allows the Hidden Gesture Recognizer to send the following kinds of gestures to the robot: • Pointing Gestures (Goto Here). Those are transferred • Direction Gestures (Move in this direction). • Rotation Gestures (Clockwise/Anticlockwise). • Speed Control Gestures (Slower/faster). • Reinforcement Gestures (Yes/No). • Stopping Gesture (Stop). • Continue what you are doing Gesture (Continue). This set of gestures was believed to cover most of the gestures of the main operator during the real experiment

The experiment was done by 22 main operators. Only 15 of these subjects will be used in the analysis in this paper because they are the only ones who completed the pre-questionnaire. The same person served as the Hidden Gesture Recognizer in all the sessions. Each subject interacted with the robot using the same map in three episodes of the game. The robot used one of the following feedback modalities for every session: 1) Stop: In this case, the robot just stops whenever it needs to give a feedback. 2) Speech: In this case, the robot gives a predefined statement for every kind of feedback signal. 3) Motion In this case, the robot gives a motion feedback. The order of the modalities was shuffled to counter any correlation between the order and the results. Before beginning the experiment the main operator was asked to report their expectation of the robot’s intelligence in a 0-10 scale. After finishing interacting with each modality, the main operator guessed how many objects of each kind were existing, and ranks the robot in terms of its ability to recognize gesture, the understandability of its feedback, and how efficient it was in giving this feedback in a 010 scale. The main operator is also asked to rank how enjoyable was the session in the same scale. An extra question about the quality of the sound is added in the Speech modality. The Hidden Gesture Recognizer also ranked his own gesture recognition accuracy after each episode (0-10). After finishing the whole set, the main operator is asked to select his/her preferred feedback modality and the fastest episode, and to write an optional free comment. The results of those sessions were statistically analyzed to determine the patterns of gesture use in the interaction and to measure the effectiveness of each feedback modality. The following section details the study of human adaptation as manifested in the gestures used by various subjects. For details about the modality analysis refer to [10]. IV. R ESULTS AND D ISCUSSION The first type of adaptation behavior studied was the evolution of the use unknown gestures with time. Fig. 4 shows this evolution for the whole data set, for participants with high expectation (defined as selecting 5 or more on the scale of expected intelligence in the pre-questionnaire), and for participants with low expectation (3 or less on the scale of expected intelligence). To control the effect of different time periods for different sessions, the three session of every participant were concatenated and then quantized into ten time slots and the analysis was done according to those time slots rather than the actual time. The figure shows a tendency to reduce the number of unknown gestures used with time in all cases which indicates that the subjects were able to limit their gestures to the kinds understood by the robot (this corresponds to the replace behavior in [9]). Detailed analysis of the figure shows that this adaptation behavior is

Fig. 4: Evolution of the average number of unknown gestures used with time. Every session is divided into ten time slots of equal length

not linear and is dependent on the pre-expectation of the subject. In all cases adaptation did not start immediately but after a time that depends on the expectation. Subjects with high expectation tended to use more unknown gestures in the first quarter of the interaction, then their use of unknown gestures dropped sharply with time until it ended up to the same value as subjects with low expectation. It is also visible that subjects with high expectation tended to start this kind of adaptation to the robot after a longer period of trying unknown gestures compared with subjects with low expectation. Kendall’s τb correlation coefficient (2 tailed) was .438 with significance of 0.015 which is significant at the 0.05 level. The Spearman’s R2 correlation coefficient (2 tailed) was 0.573 with significance 0.002. Pearson correlation coefficient (2 tailed) was 0.75 with significance less than 0.001. This quantitative analysis indicates the same results visible from Fig. 4. To control the robot, the participant had to specify the direction of movement, the speed and duration. The set of gestures available to the Gesture Transfer software includes three kinds of gestures to control the direction of movement: • •



Direction Gestures which are understood by the robot as rotation commands to a specific angle. Rotation Gestures which are understood as rotation commands with only specific direction (clockwise or counterclockwise). Pointing Gestures: These gestures are understood as GoTo commands to the target of the gesture. There were two implementations of the response to this gesture: In the navigation capable case the robot rotates to the direction of the pointing target and moves to it in a straight line unless an object appears in its way slowing down when approaching the target point. In the navigation incapable case the robot just rotates to the direction of the target point and continues whatever navigation act is was doing before (e.g. going forward, stopping, etc).

When the subject has higher expectation from the robot,

Fig. 5: Distribution of Expectation, Relative Use of Pointing Gestures, and Use of Unknown Gestures over the Main Operators

the use or pointing gestures relative to other more primitive direction control gestures is expected to increase. To confirm this effect we used the normalized pointing measure which is defined as the ratio between the number of pointing gestures to the total number of direction and rotation gestures. Higher normalized pointing values indicate that the subject is assigning more autonomy to the robot than lower values. TABLE I: Analysis of correlation between the main operator’s expectation, the number of unknown gestures, and the normalized pointing measure in early and late stages of the interaction. Test

Early Late Exp. NP UG NP UG Kendall 1.0 .60∗∗ .44∗ -.28 .288 Exp. Spearman 1.0 .72∗∗ .57∗ -.35 .343 Pearson 1.0 .79∗∗ .75∗∗ -.36 .344 Early Kendall .60∗∗ 1.0 .333 NP Spearman .72∗∗ 1.0 .441 Pearson .79∗∗ 1.0 .74∗∗ Late Kendall -.28 1.0 -.061 NP Spearman .343 1.0 -.08 Pearson .344 1.0 -.177 ∗∗ Significant at level 0.01 (2 tailed) ∗ Significant at level 0.05 (2 tailed) NP: Normalized Pointing Measure UG: Number of Unknown Gestures Exp.: Pre-Expectation of the main operator Early: Based on the first 20% of interaction time Late: Based on the last 20% of interaction time - : Not informative

To study the relation between the expectation of the subject and his use of gestures, we scaled the expectation, unknown gesture use, and normalized pointing to the range 0 1. Fig. 5 shows the relation between the scaled versions of

Fig. 7: Evolution of the Normalized Pointing Measure with time according to the type of navigation behavior of the robot

Fig. 6: Evolution of the use of pointing, direction & rotation, and normalized pointing gestures with time according to the expectation of the participant

these three measures. As the figure shows there is high correlation between expectation and both unknown gesture use (as discussed previously) and normalized pointing. Kendall’s τb correlation coefficient between normalized pointing and the expectation was 0.597 with significance of 0.002. Spearman’s R2 correlation coefficient was 0.724 with significance of 0.001 while Pearson correlation coefficient was 0.792 with significance less than 0.001 (see Table I). These results indicate that the normalized pointing measure is indeed highly correlated with the expectation of the user. Detailed analysis of the evolution of various kinds of navigation control gestures is shown in Fig. 6. As the figure shows there is a constant trend to reduce the use of pointing gestures with time. One reason for this tendency lies in the task design itself as using pointing gestures corresponds to moving the robot along straight lines while the maps used in the experiment were using smooth curves (see Fig. 3). Nevertheless the adaptation effect is very visible. For example the average number of pointing gestures used by users with high expectations was 5.27 times the number of direction and rotation gestures in the first tenth of the interaction time compared with 1.09 for subjects with low expectation. By the end of the interaction a completely different picture

appears where the average number of pointing gestures for subjects with high expectations is less than 0.25 the number of direction and rotation gestures compared with 0.56 for subjects with low expectation. At this last tenth of the interaction Kendall’s τb , Spearman’s R2 and Pearson correlation coefficients between the normalized pointing measure and the initial expectation shows no significant correlation (see Table I). This result can be interpreted that the subjects adapted their use of gesture to the task which requires more control over the behavior of the robot (due to the smoothness of the maps) than what the straight line navigation behavior of the robot can achieve. To discover if the users also adapted their behavior to the robot capabilities concerning the use of pointing gestures; correlation between the change in the normalized pointing measure and the kind of navigation behavior executed by the robot is measured. Fig. 7 shows the percentage of the normalized pointing gestures used in every time slot for the two different implementations of the GoTo behavior invoked by the pointing gesture. As the figure shows although they started from roughly the same point, the users tended to use more pointing gestures toward the end of the interaction in the navigation capable case. One explanation of that is that the users were able to adapt their behavior to the capability of the robot not only the task. It is clear from the figure that the distribution is not linear. This can be explained by the fact that in the beginning of the interaction adaptation to the task wins over adaptation to the robot capabilities (this is also congruent with the low adaptation to the robot in the interaction early stages discovered by analyzing the use of unknown gestures). As the interaction goes, the main operator becomes more aware of the capabilities of the robot and more capable of adapting to it. V. C ONCLUSION The results presented in the previous section (if confirmed by larger scale experiments) reveal some important aspects

of human adaptation to miniature robots in collaborative navigation situations: 1) The main subjects did not start adapting to the robot from the very beginning to the interaction and their rate of adaptation is not fixed along the interaction time. This can be explained by the existence of an exploration phase in the beginning of the interaction during which the robot’s behavior should be consistent to allow the human to get a feel about its capabilities and help him/her adapting to it. By the end of the interaction there is another period of low adaptation rate that can be explained in this experiment by the fact that the robot was not adapting its behavior to the human which made it possible to the human to completely discover the capabilities of the robot regarding the task and so saturates his/her adaptation. 2) The main subjects showed adaptation behavior that depends on the expectation they had from the robot. Adaptation started later for subjects of higher expectation compared with subjects with lower expectations. This stresses the importance of controlling the user expectations if successful mutual adaptation is to be achieved. Over expectation can cause reduced adaptability of the human to the robot while underexpectation can reduce the space of behaviors that the human is exploring in the beginning of the interaction. 3) The main subjects showed signs of task adaptation earlier than signs of adaptation to the robot capabilities. Although this can be task dependent but at least in similar tasks it suggests again a period of low adaptation from the side of the robot in the beginning of the interaction which should depend on the complexity of the task at hand. Future directions of research in this area includes using the results of this experiment to build an autonomous robot that can achieve mutual adaptation with the human by respecting the three properties of human adaptation discovered in this study. An interesting question regarding this future study is whether or not the period of saturated adaptation will appear and if there is a way to get the interaction out of it once happened. Another interesting question is how can the human adaptation to the robot (especially the period of low adaptation in the starting of the interaction) be affected by the complexity of the task and the distribution of the knowledge regarding it between the robot and the human. Another interesting question is the effect of the feedback modality used by the robot on the adaptation behavior of the human. This point was difficult to analyze in this experiment because the order of the sessions is expected to have more effect on the adaptability of the human than the feedback modality. A future inter-subject design can be useful in analyzing this point. R EFERENCES [1] Y. XU, K. UEDA, T. KOMATSU, T. OKADOME, T. HATTORI, Y. SUMI, and T. NISHIDA, “Woz experiments for understanding mutual adaptation,” Journal of AI and Society, 2007.

[2] T. Ogata, S. Sugano, and J. Tani, “Open-end human robot interaction from the dynamical systems perspective: mutual adaptation and incremental learning,” in IEA/AIE’2004: Proceedings of the 17th international conference on Innovations in applied artificial intelligence. Springer Springer Verlag Inc, 2004, pp. 435–444. [3] M. Imai, H. Kawasima, and Y. Honda, “Generating behavioral protocol for human-robot physical-contact interaction,” in ICRA 2005: IEEE International Workshop on Robots and Human Interactive Communication, 2005, pp. 229–234. [4] Y. F. O. Mohammad and T. Nishida, “Towards combining autonomy and interactivity for social robots,” AI and Society Journal: Special Issue about SID 2007, 2008, in press. [5] P. Rani and N. Sarkar, “Operator engagement detection and robot behavior adaptation in human-robot interaction,” in the 2005 IEEE International Conference on Robotics and Automation, 2005, pp. 2051–2056. [6] A. L. Thomaz, “Socially guidedmachine learning,” Ph.D. dissertation, MASSACHUSETTS INSTITUTE OF TECHNOLOGY, May 2006. [7] E. Martinson and D. Brock, “Improving human-robot interaction through adaptation to the auditory scene,” in HRI ’07: Proceeding of the ACM/IEEE international conference on Human-robot interaction. New York, NY, USA: ACM, 2007, pp. 113–120. [8] Jayawardena, Chandimal, Watanabe, Keigo, Izumi, and Kiyotaka, “Controlling a robot manipulator with fuzzy voice commands using a probabilistic neural network,” Neural Computing & Applications, vol. 16, no. 2, pp. 155–166, February 2007. [Online]. Available: http://dx.doi.org/10.1007/s00521-006-0056-8 [9] Y. Xu, M. Guillmot, and T. Nishida, “An experiment study of gesturebased human-robot interface,” in IEEE/ICME International Conference on Complex Medical Engineering-CME2007, 2007, pp. 458–464. [10] Y. F. O. Mohammad and T. Nishida, “Talkback: Feedback from a miniature robot,” in Twentieth Australian Joint Conference on Artificial Intelligence, December 2007, pp. 357–366.