Keywords: personified interface agents, persona effect, empirical evaluation. ..... Table 3: Means for the Persona-Specific Questions asked in the Questionnaire ...
The Persona Effect: How Substantial Is It? Susanne van Mulken, Elisabeth Andre & Jochen Millier German Research Centre for Artificial Intelligence (DFKI), Stuhlsatzenhausweg 3, D-66123 Saarbrucken, Germany. Tel: +496814128, +496815252 EMail: {mulken.andre.jmueller}@dfki.de Personification of interface agents has been speculated to have several advantages, such as a positive effect on agent credibility and on the perception of learning experience. However, important questions less often addressed so far are what effect personification has on more objective measures, such as comprehension and recall, and furthermore, under what circumstances this effect (if any) occurs. We performed an empirical study with adult participants to examine the effect of the Ppp Persona not only on subjective but also on objective measures. In addition, we tested it both with technical and non-technical domain information. The results of the study indicate that the data from the subjective measures support the so called persona effect for the technical information but not for non-technical information. With regard to the objective measures, however, neither a positive nor a negative effect could be found. Implications for software development are discussed. Keywords: personified interface agents, persona effect, empirical evaluation.
1 Introduction In the past years, there has been a growing interest in the personification of intelligent interface agents. Personified agents are interface assistants that are visually present on the computer screen in the form of an animated life-like character (Ball et aI., 1997; Rickel & Johnson, 1997; Rist et aI., 1997). Their tasks range from merely performing the tasks delegated to them by the user (e.g. collect particular information from the World-Wide Web) to explaining and presenting information on the screen.
H. Johnson et al. (eds.), People and Computers XIII © Springer-Verlag London 1998
54
Susanne van Mulken, Elisabeth Andre & Jochen Maller
The main reason for personification is that it is believed to render humancomputer interaction more human-human like, more social (Nass et al., 1994). This is speculated to lead to several advantages: • The social aspect of personified agents is believed to raise the trustworthiness and believability of agents - aspects which are important if interface agents are to be authorized by the user (Lester & Stone, 1997) . • Because it makes interaction more social, it increases the user's engagement (Walker et al., 1994; Sproull et aI., 1996). Furthermore, in learning environments, personification can positively affect the student's perception of his learning experience (Lester et al., 1997a). However, personification of intelligent interface agents not only offers opportunities, it also raises questions. Many empirical evaluation studies that have been carried out so far have started from the assumption that personification should indeed be strived for, the only question to answer is how this should be done. Furthermore, most studies have concentrated on subjective measures such as acceptance and believability (Takeuchi & Naito, 1995; Koda & Maes, 1996). An exception to this forms a study by Lester et al. (1997b) who investigated the effect of different clones of an interface agent exhibiting different types of feedback behaviour on learning performance in a learning environment for children. However, the main question addressed in this study was what type of feedback the agent should exhibit and not what effect the mere presence of such an agent has on learning performance. A recent informal study by Lester and colleagues (Towns et al., 1998) concentrated on the empirical evaluation of a Persona's pointing gestures. However, the study only investigated whether the Persona's referential expressions were intelligible; it did not compare the learning effect in the Persona condition with the learning effect in a no-Persona condition. To investigate this issue, we performed an explorative empirical study that evaluates the effect of one such agent, the Ppp Persona (Rist et aI., 1997) - both on subjective and on objective measures. The Ppp Persona is a life-like interface agent that presents multimedia material to the user following the directives of a script (Andre & Rist, 1996). This material is either automatically generated or retrieved from the World-Wide Web. While the user views the presentation, the Persona can among other things comment on particular parts and highlight them through pointing gestures. Currently, the repertoire of presentation gestures includes: gestures that express emotions (e.g. approval or disapproval), gestures that convey the communicative function of an utterance (e.g. to warn, to recommend, or to dissuade), gestures that support referential acts (e.g. to look at an object and point at it), gestures that regulate the interaction between the Persona and the user (e.g. to establish eye-contact with the user when talking to him or her), and gestures that indicate that the Persona is speaking·. However, the Persona's behaviour is not only determined by the directives specified in the script. Rather, the behaviour of the animated character follows the equation: • In the experiment, however, these behaviours were not fully put to use.
The Persona Effect: How Substantial Is It?
55
Persona behaviour = directives + self-behaviour Such self-behaviours are indispensable in order to increase the Persona's vividness and believability. Currently, they comprise idle-time actions, such as tapping with a foot, actions for indicating activity, such as, to turn over pages in a book, navigation acts, such as walking or jumping, and immediate reactions to external events, such as mouse gestures on the presented material. While the presentation script is created in a proactive planning phase, the Persona's selfbehaviours are determined reactively taking into account the current situation at runtime. In the experiment we performed, we were specifically interested in: • The effect of the presence of Persona on a comprehension and recall task and its effect on subjective assessment. • Any effect of the domain of the presented information on these measures. We believed three possible effects on comprehension and recall to be possible: • Persona increases comprehension/recall performance in the user, for instance, through an increase in the user's motivation; • Persona decreases performance on comprehension and recall, for instance, because it attracts the user's attention to the Persona herself and thus away from the relevant information; • Persona neither decreases nor increases comprehension and recall, for instance, because neither of the above mentioned effects exists or because the respective effects are equally strong and thus - acting as antagonistic factors - prevent any effects on comprehension and recall from being observable. On the basis of related studies, we expected that, with respect to subjective assessment, presentations with Persona should be rated more positively (with respect to such issues as entertainment and interestingness) than presentations without Persona. The exploratory study consisted of three parts: In the first two parts, we investigated the effect of Persona on the objective measures with respect to technical and non-technical information. In the last part, we were interested in the influence of Persona on subjective assessments of the presentation.
2 Method
2.1
Subjects and Design
Subjects were 15 females and 15 males, all native speakers of German, on average 28 years of age, and recruited from the Saarbriicken university campus. The subjects were paid DM 10,- for participation. The experiment was conducted in German and lasted about 45 minutes. The independent variables were Persona (with the levels absent and present) and the Type of Information (technical vs. non-technical information). As dependent
Susanne van Mulken, Elisabeth Andre & Jochen Maller
56 of I Type Information
Persona
Technical Non-technical
Present
Absent
5s 1-15 Comprehension! Recall! Quest. A+B 5s 1-15 Comprehension! Recall! Quest. A+B
5s 16-30 Comprehension! Recall! Quest. A 5s 16-30 Comprehension! Recall! Quest. A
Figure 1: Schematic representation of the experimental design.
Figure 2: Pulley systems used for the technical presentations.
variables, we defined perfonnance on comprehension and recall tests and subjective ratings on an assessment questionnaire. The variable Persona was manipulated between-subjects, whereas the variable Type of Infonnation was manipulated within-subjects (see Figure 1). All subjects were assigned to the conditions randomly. Twenty-one of these subjects had no prior familiarity with the building in which the experiment was carried out. Of the nine remaining subjects, four were in the no-persona condition, five were in the persona condition.
2.2 Materials 2.2.1
Presentations
In both Persona conditions, subjects were presented with technical and non-technical material. For the technical material, we relied on the material used by Hegarty et al. (1988) and Hegarty & Just (1993) to investigate the understanding of machines from
57
The PersoTUJ Effect: How Substantial Is It?
Jala IIeda!r Redmenwu1W11
0.16 0.17 0.18 0.19 0.20 0.21 0.22 0.23 0.24
0.15 0.14 0.13 0.12
p.ll 0.10 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02
Figure 3: Example of the items used in the non-technical presentations. For privacy reasons, we here substituted the photograph with that of one of the authors.
text and diagrams. The material consisted of information about four different pulley systems (see Figure 2). The spatial information concerning each of the pulleys was presented visually through its depiction on the screen. Whereas in Hegarty & Just (1993), information about $e pulley system's parts and its kinematics was presented textually, in this experiment, it was presented acoustically so as to enhance Persona's life-likeness. For the condition with non-technical material, we designed a presentation in which ten fictitious employees of DFKl were introduced. For each employee, his or her photograph was presented. In addition, the employee was introduced by hislher name and occupation. The office which slhe worked in was pointed at in a map of the DFKl floor (see Figure 3 for example material). The occupations were kept as distinct and concrete as possible. Examples are press officer, secretary, and programmer. Moreover, so as to ease discrimination, the offices assigned to the employees were never adjacent.
2.2.2
Tests
Subjects were tested on their comprehension of the pulley system material in four test sessions, each directly following the presentations. Test questions concerned both
58
Susanne van Mulken, Elisabeth Andre & Jochen Maller General Questions (Q-A) 1. Did you find the presentation difficult to understand?
2. Did you find the presentation entertaining? 3. Did you find the tests difficult? 4. Did you find the presentations interesting? 5. Did you feel overloaded by the infonnation presented? 6. General remarks about the presentations: ... Persona-8peclfic Questions (Q-B)
7. Was Persona's behaviour appropriate for the presentation? 8. Did Persona distract you from the relevant infonnation? 9. In which of both parts did you find Persona more appropriate? 10. Did Persona help you concentrate on relevant infonnation? 11. Did Persona motivate you to further pay attention to the presentation? 12. If in future you could choose between presentations with and without Persona (but with arrow annotations), which would you prefer? 13. Other remarks about Persona: ...
Figure 4: Questionnaires A and B. Where possible, questions were to be answered on a fivepoint scale, for the pulley system material and the DFKI-employee material separately.
configuration and kinematics of the pulley systems. An example of the questions about the configuration is "What objects does the red rope touch?". An example of the kinematics questions is "How does the lower pulley move if one pulls the free end of the red rope?". The former type of questions was open-ended. The latter type of questions was multiple choice: Possible answers to the kinematics questions were, for instance, "Does it move up, down, or stay in the same place", and "Does it rotate clockwise, counter-clockwise, or not rotate?". It was assumed that a prerequisite for correct answers to these questions is that the subject had built a mental model of the system, which in tum is regarded as indicative of understanding (Weidenmann, 1988). The recall tests on the employee material followed after all employees had been introduced. On presentation of an employee's photograph and a map of the DFKI floor showing numbered rooms, subjects had to fill in the employee's name, occupation, and office number. At the end of the experiment, the subjects were asked to fill in a questionnaire. This questionnaire consisted of two parts (Q-A and Q-B). The first part contained questions that concerned, among other things, the difficulty of the presentation and the tests, and the degree of entertainment experienced during the presentations (see Figure 4). The second part of the questionnaire (Q-B) contained questions concerning Persona. This part aimed to find out any differences in acceptance depending on the type of information presented. An example of these questions is "Did you find Persona's behaviour appropriate for the presentation?". These questions were to be answered on a five point scale for both domains. Whereas the first part, Q-A, was
The Persona Effect: How Substantia/Is It?
59
Figure 5: Example presentations for the technical domain information. Left: No-Persona condition; Right: Persona condition.
This pulley system consists of 3 pulleys (
