A Usability Evaluation Method for Virtual Reality User Interfaces Sutcliffe A.G. and Deol Kaur K. Centre for HCI Design Department of Computation UMIST PO Box 88 Mancheser M60 1QD
[email protected] +44-161-200-3315
ABSTRACT A walkthrough method for evaluating virtual reality (VR) user interfaces is described and illustrated with a usability assessment of a virtual business park application. The method is based on a theory of interaction that extends Norman’s (1986) model of action. A walkthrough analysis method uses three models derived from the theory. The first model describes goal-oriented task action, the second exploration and navigation in virtual worlds, while the third covers interaction in response to system initiative. Each stage of the model is associated with generic design properties that specify the necessary support from the system for successful interaction. The evaluation method consists of a checklist of questions using the properties and following the model cycle. Use of the method uncovered several usability problems. Approaches to evaluation of VR applications and future work are discussed. KEY WORDS Usability evaluation, walkthrough method, virtual reality
1. INTRODUCTION Virtual Reality (VR) systems can suffer from severe usability problems such as conceptual disorientation and inability to manipulate objects (Kaur et al, 1996); however, no methods for evaluating the usability of VR systems have been reported. There is a need for better-designed VR systems (Bolas, 1994, Loeffler and Anderson 1994) that support perception, navigation, exploration and engagement (Wann & Mon-Williams, 1996). Significant usability problems with current VR systems have been reported by Miller (1994); furthermore, Kaur et al (1996) in a field study of design practice found that designers lacked a coherent approach to design, especially interaction design, lacked understanding of usability concepts underlying VR and did not use conventional HCI methods or guidelines. Standard evaluation methods (e.g. Nielsen, 1993) may be able to discover some usability problems but, as Hook and Dahlback (1992) note, no current evaluation methods fit the specific problem of VR applications. It may be argued that conventional usability evaluation methods, such as heuristic evaluation (Nielsen, 1993), or cooperative evaluation with users to diagnose problems (Monk et al, 1993) or cognitive walkthrough (Wharton et al, 1994) could be applied to virtual reality systems. However, Nielsen’s heuristics for instance do not address issues of locating and manipulating objects, or navigating in 3D worlds; while cognitive walkthroughs (Wharton et al, 1994) were not designed to address perceptual orientation and navigation and interpreting change in virtual worlds. Usability lab style evaluation (Monk et al, 1993) may identify breakdowns and problems in interaction but these methods provide little guidance towards a solution. Therefore, there is a need to support the process of usability evaluation that addresses the new problem posed by VR as well as linking identification of problems to specific interface design guidelines. The link from problem diagnosis design advice in many usability evaluation methods is weak. In our previous work we have extended cognitive walkthrough to address evaluation of display based interfaces (Sutcliffe et al, in press). This provides a starting point to investigate the problems of VR evaluation. In this paper we describe a formative evaluation method based on a theory of interaction for virtual reality and complex 3D graphical user interfaces. The method is primarily aimed at desktop VR systems, although some of the recommendations may be applicable to immersive VR. As desktop VR constitutes the majority of current industry applications (Kaur et al, 1996), we decided to leave immersive VR evaluation to the second phase of our research. The paper is structured in four sections.
1
First we describe the model of interaction that is derived from the theoretical research. This is followed by a section which covers the walkthrough evaluation method derived from the theory. Each stage of the method is illustrated by a case study using a Business Park exploration/marketing application. The paper concludes with a discussion of related work.
2. MODEL OF INTERACTION The approach we take to evaluation follows the cognitive walkthrough tradition of Lewis et al (1990). Our starting point is based on an extension of Norman’s (1986) model of interaction that was used as the underpinning for earlier cognitive walkthrough methods. Since virtual reality poses new problems of navigation, orientation and movement for user interaction, we have extended Norman’s model to describe the user’s mental and physical actions in 3D worlds. An overview of the interaction theory for virtual environments with further details of its validation is reported in Kaur (1998). The theory proposes a model composed of three cycles. The most important cycle is task based activity, when the user is trying to achieve a goal. Subordinate to this cycle is navigation/exploration when the user’s intention is simply to find their way around the virtual environment. In some cases exploration may become the task goal in its own right. We distinguish between goal directed exploration and serendipitous browsing in which the user’s goal is not clearly formed. The third cycle describes system initiative when the user’s role is to interpret actions taken by the system and respond to them when initiative is returned, or interrupt and seize initiative from the system if necessary. System initiative may be a manifestation of other user action via avatars in collaborative virtual environments; however, conversational control and multi-party interaction is not within the scope of the model. Figure 1. VR interaction model for the task action cycle (after Norman, 1986). The nodes describe the user’s cognitive actions and the arcs show the possible pathways between actions. The cycle starts with goal formation (step 1 in figure 1) in which the user forms an intention to achieve a task goal or sub-goal and retrieves the appropriate procedure from memory. Procedures are scripts of actions that must be carried out to achieve the task goal, so the user will commence interaction with an intention to carry out the first action in the procedure. The first problem to be solved is whether the immediate environment contains the necessary objects for interaction. If the objects implied in the first task action are not visible then the user has to search the environment for them (step 2). This leads into the navigation sub-cycle and exploration until the necessary objects are found or the user gives up. Assuming that a familiar part of the virtual environment is found and known to contain the necessary objects, then the user has to locate the objects (step 3), approach and orient the representation of self in the correct position for interaction (step 4). With simple and larger objects these steps will not be necessary, as depicted by the dashed line that by-passes them. Approaching and orienting to objects may be a concurrent activity with ‘specifying action - step 5’, deciding what action to carry out. The model does not distinguish between concurrent or sequential action, so the system should support user activity whatever its ordering. Specifying action is followed by manipulating objects and executing actions (step 6). Actions should be followed by feedback from the environment. In continuous actions manipulation and feedback are interleaved as the user may adjust action in light of feedback. First the user must receive and recognise the feedback (step 7) and then interpret the nature of the change (step 8). Once an action is complete the user may continue with specification of the next action by loading the next action from procedural memory (step 9). This will usually lead to approach and orientation, unless the necessary objects are not visible in which case the sequence forks to step 2. Alternatively if completion of the action has reached the end of a procedure then the user proceeds to the next sub-goal and the cycle resumes (step 1). However, if interaction causes change in the VE which obscures the necessary objects, then the user may have to re-orient and approach again. Specification of action may be determined by retrieving task procedures from memory or by problem solving in response to the environment, i.e. situated action (Suchman, 1987). The main connotation of the procedure driven versus situated action is that the environmental needs suggest action more directly in the latter. Since there is considerable evidence that users specify action using environment cues (Reiman & Young, 1996; Kitajima & Polson, 1997) in graphical user interfaces, this is likely to be the dominant mode in VEs.
2
System initiative may occur at any point in the task-action cycle. The behaviours in this sub-cycle (see figure 2) are first realising that the system has taken the initiative (step 1), although this is only relevant if the system has taken control as in a guided tour; alternatively, the cycle starts by recognising and interpreting change (steps 2, 3). The user then has a choice (step 4) whether to allow the system to retain initiative until the action sequence terminates, or to interrupt. If the former is chosen then the observe-interpret loop continues until the system action ceases, when the user plans a response and carries it out (step 5). In the latter the user decides when to interrupt and regain the initiative. Further action may then transfer back to the task action cycle, although in conversational, groupware applications interaction may be an interlinked set of conversational turns between different users, each taking the system initiative via a mannequin representation. Figure 2. Interaction model for user behaviour in response to system initiative (i.e. action by an avatar in a virtual environment) The system exploration sub-cycle, illustrated in figure 3, has two main paths. In serendipitous exploration when the user has no specific goal in mind, interaction is composed of a series of scan environment/decide direction (step 2) - navigate (step 3) - interpret change (step 4) loops. If interesting objects or information are encountered these may be memorised for subsequent use (step 5) or may be explored further by manipulation, which is modelled as goal directed exploration in the task action cycle. In goal directed exploration, the user decides what to find and then forms a plan about how to find it (step 1). If the environment affords no clues about where to look for the search target, the user may resort to guesswork. Interaction proceeds by a loop of scan/decide direction - navigate - interpret actions, although in this case the interpretation is whether the user has found or is closer to their goal. Exploration motivated by the task-action cycle reverts back to the main cycle when the appropriate part of the environment is found. In many applications, goal directed exploration is the main user task, so interaction may reside primarily within this sub-cycle. Figure 3. Interaction model for user exploration and navigation in virtual environments. During interaction the user’s behaviour will interleave this model and figure 1 as bouts of exploration and goal direction action occur. The models of interaction are general descriptions of how user action should occur for a novice user. With expert and skilled users, stages in the cycle will become increasingly automated, so navigation and orientation may be reduced and interaction becomes a closely coupled loop of action interleaved with feedback. The models of interaction are used to drive the walkthrough questioning dialogues, that are elaborated in the following sections.
3. WALKTHROUGH EVALUATION METHOD The walkthrough is divided into the main task-action cycle with navigation and system initiative subcycles. The method is illustrated by a case study evaluation of a Virtual Reality Business Park application, illustrated in figure 4. Before the walkthrough can commence it is necessary to establish representative tasks that users will be asked to perform. Figure 4. VR application for explaining facilities in a business park, courtesy of the Welsh Rural development agency. The walkthrough starts with a preparation phase. This is composed of task/domain analysis and creating a user profile. This is followed by the walkthrough which follows the task-action or exploration path script. The method tests user interfaces before they are fully implemented. It can be carried out on specifications, storyboards, and early prototypes. The evaluation can be carried out by personnel who are not HCI experts although training in HCI will improve the accuracy of the usability diagnosis. Repeated analysis with two or more evaluators will also improve the coverage and accuracy of the results, as found in other usability evaluation methods (Nielsen, 1993). The walkthrough is organised by posing questions that focus on usability problems cross references to the GDPs that help problem diagnosis according to the stage within each cycle of interaction. The walkthrough procedure is: • Prepare scenarios that represents the user’s tasks as goals and actions to be carried out, general
3
• • •
•
exploration and navigation, and reaction to system actions. Walk through each scenario using the appropriate model (figure 2 1-3), asking the questions detailed in the following sections 3.1-3.3. Critique the design using the question and the GDPs linked to the action stage in tables 1-3; record problems in terms of missing GDPs or VE design features. Continue walkthrough to the end of the scenario script and having tested all the scenarios, collate the indicated problems into a list of missing GDPs linked to VE design features where possible, and indicate missing features. Prioritise problems by subjective judgement of their severity.
3.1 NORMAL TASK ACTION To achieve successful interaction the user requires knowledge about the domain and the environment on one hand and affordances provided by the machine on the other. Table 1 shows the stages of interaction with the knowledge sources and affordances, which we term ‘generic design properties’ (GDPs). The task action model (see figure 1) provide a means of interpreting the context of interaction either when user errors are observed or to prompt evaluation questions for each stage of interaction as recommended in cognitive walkthroughs for standard user interfaces (Wharton et al, 1994). GDPs express abstract requirements in a design to assure successful interaction, but they need to be specialised into concrete design guidelines. Specialisation of GDPs has been described in Kaur et al (1999), and space limitation preclude an extensive description in this paper. Table 1. Actions in the task cycle cross reference to walkthrough questions, GDPs and user knowledge for successful interaction. Questions are directed to elicit potential user problems, then the answers are cross-referenced to the GDPs that indicate possible causes of usability problems. The evaluator steps through the task for each sub-goal in the sequence, progressing around the action cycle illustrated in figure 1, with the following questions: (i) Can the user form or remember the task goal? The answer to this question will be yes unless the user has poor task knowledge. If this is the case an aide memoir of the task may be displayed as a bullet point list of the steps; otherwise, training in the task should be provided. The goal formation stage requires user knowledge of the task, while locating the active part of the virtual environment (VE) requires cues about the whereabouts of the appropriate objects (GDP 2). These may be delivered by the environment itself or by other design features, such as overview maps. The distribution and appearance of objects should correspond with the user’s expectation based on their memory of real world tasks (GDP 1). (ii) Can the user specify an intention of what to do? At this stage the user requires either a procedural memory of how to carry out the task or cues and affordances in the VE to suggest the best course of action. Affordances should be present, otherwise hints should be given about where the user might find them. (iii) Are the objects or part of the environment necessary to carry out the task-action visible? If they are not the user will have to search for them. Follow-up questions are in the exploration subcycle. (iv) Can the objects necessary for task action be located? Objects may be obscured or not visible even though the user is sure that the appropriate part of the environment has been reached. The necessary object should be highlighted or made salient. Important objects should be rendered in more detail to help recognition. If highlighting offends naturalness criteria for the environment, speech cues may be used. Locating the object requires user domain knowledge and clear cues by the system representation (GDPs 2, 5). (v) Can the users approach and orient themselves to the object so the necessary action can be carried out? Objects may be obscured, or rendered in 2D texture so depth perception is difficult. Objects should be modelled in 3D; alternatively the design of the ‘self’ may need to be improved. Changing the user’s viewpoint can help orientation, while improving the direction controls of the self can facilitate orientation actions. The same GDPs 2,5) apply for approach and orient; furthermore, affordances for actions (GDP 3) supplied by the environment are vital for this step and specification of action. (vi) Can the user decide what action to take and how? The objects may not suggest the necessary cues or affordances for action. If the user can not decide then the problem may either be a lack of detailed task knowledge or design of the virtual objects may be unclear. The detailed representation of the object should be improved. If naturalness is not vital, the
4
object can be animated to suggest actions to the user, otherwise speech or text instructions may be displayed. The self presence must fulfil expected user operations (GDP 4) so action can be planned and carried out as naturally as possible. Objects should have easily recognisable sub-components and should be approachable in that they give clear cues for orientation so the user can position the selfpresence (GDPs 6, 7) (vii) Can the user carry out the manipulation or action easily? If the action is difficult it may be beyond the physical capabilities of the users (e.g. manipulations are too precise or demand excessive perceptio-motor co-ordination); alternatively the user may not have acquired the necessary physical skill. ). Further detail is necessary for successful manipulation e.g. cues and affordances for which parts can be gripped, turned, etc. (GDPs 7, 8). If naturalness is not vital the size of the object may be scaled so it is easier to manipulate; alternatively, the necessary action can be simplified, or automated. Design of the self may need to be improved so manipulations are easier for the user to control. If the user is using a virtual tool this may need improving to make control more natural. Assessment of this step usually involves consideration of feedback. (viii) Is the consequence of the user’s action visible? Feedback may be either absent, ambiguous or hidden (i.e. it happens but is in another part of the environment outside the user’s immediate vision). Change to objects should be faithful to the real world. Modalities are important here as, ideally, feedback should include haptic as well as audio and visual representations. Remote feedback should be signaled to the user and its location shown. If feedback is not clear the object may be highlighted to denote change. If force feedback is not possible, cross modal substitution may be used, e.g. tone in sound or change in colour to represent pressure. Text and speech may be necessary to explain complex changes. Once action is being carried out the objects in the environment and their features must be easy to operate within the normal bounds of human physio-motor abilities. This implicates design not only of the objects being acted upon but also the representation of the self and possibly a virtual tool. All three have to be easy to operate for interaction to be successful. (ix) Can the user interpret the change? Unless the naturalness principle is offended, feedback should always be clear and unambiguous. The user should be able to interpret the effect in light of their task/domain knowledge and the relationship between the effect and the observable VE. If the effect of change is not clear, the feedback may need to be clarified or possibly explained to the user. Explanation agents may be necessary for complex effects; alternatively the overall effect may be shown in slow motion to aid interpretation. Moreover feedback, which is recognisable and easy to interpret (GDPs 9, 10) is an important ingredient of success. Operations might be visually apparent but without appropriate force feedback the extent and limits of manipulations are often difficult to gauge. (x) Can the user decide what to do next? At this stage the pathway branches. If the user has completed a task procedure then the next stage is to form the next goal, so repeat the analysis starting with (i). Alternatively if the user is within a procedure the next step is to select the next action to perform. Failure at this step may be caused by memory failure or inadequate user task knowledge; however, failure may also be due to a misleading virtual environment which gives inappropriate cues. In this case the environment needs to be redesigned to suggest the next action so that it is compatible with the user’s task. Note that this step may be related to iterations between questions (vi) and (vii) for closely coupled and continuous actions. When the user is skilled, deciding the next action is automatic. Once the next action has been selected the user may re-enter the cycle at question (iii), (iv) or (v) depending on where the necessary objects are located. 3.2 GOAL DIRECTED EXPLORATION The exploration and system initiative cycles have similar support implications, although these are oriented towards clear cues from the environment for navigation and communication by system agents. In goal directed exploration the user has a definite search target, motivated either by the task or by the need to explore specific aspects of the environment. Entry to this sub-cycle assumes that the search goal set by the task action cycle indicates a need to locate a specific part of the virtual environment; alternatively the user may set their own goal to explore the VE The generic design properties required in the exploration cycle are similar, although more support for orientation is necessary. Table 2. Actions in the exploration/navigate cycle cross referenced to walkthrough questions, GDPs and user knowledge.
5
The walkthrough questions that follow are: (i) Does the user know where to start looking? If the user has poor knowledge of the virtual environment or the environment gives no clues about where appropriate objects may be located, then search will be by guesswork. Either familiarisation of the virtual world should be provided by guided tours, or overview maps displayed to show the extent of the VE and an outline of its contents. In some applications with extensive virtual worlds, a search facility may be necessary so the user can enter and the system then takes the user to that point. The environment should have a clear structure and pathways unless the domain implies otherwise, e.g. a VE for training mariners (Darken & Sibert, 1996) to navigate at sea. (ii) Can the user determine a pathway towards the search target? The system should have a clear structure so directions for movement are obvious. Other ways of remedying user problems at this step are to give overview maps of the VE and trace the appropriate path to a search target indicated by the user. Providing a means for the user to leave aide-memoir waymarks as cues to ‘where I have been’ (GDP 8) and salient landmarks in the VE can help memorisation of paths and locations, while allowing the user to change the viewpoint can help provide a fresh perspective to find pathways. The environment should contain orientation cues for navigation (GDP 3) while the self-presence needs clear controls for direction, speed, yaw and pitch. (iii) Can the user execute movement and navigation actions? Poor design of the self-representation may cause problems at this step. For instance, if the user does not know the gestures for movement with a hand/dataglove presence, then training should be provided or an animated demonstration given. Specification of power actions (e.g. fly-through, rapid movement) should be explained to users. (iv) Can the user recognise the search target? The search target may be obscured or not clear. The design should represent the desired objects in detail with a clear outline to enhance recognition. If the user’s search target can be communicated to the system then it can be highlighted. When naturalness prevents highlighting, magnification facilities may be useful to help users identify small targets. Objects should have clear components and reflect change in the user’s viewpoint (GDP 6, 7) by use of shading from directional light sources, and faithful depth perspective. (v) Can the user find a location of interest again ? Waymarking or pathway trace facilities may be needed to help the user’s navigation (GDP 8). Once the user has found and recognised the search target the walkthrough re-enters the task action cycle at question (iv). 3.3 EXPLORATORY BROWSING In this case exploration the user’s aim is to explore the system for the sake of curiosity. There is no specific goal; however, various objects may need to be investigated and remembered for future reference. The walkthrough questions follow an iteration of decide direction-navigate-interpret actions (2, 3, 4 in figure 2) and use the same sub set of table 2. (i) Can the user determine a pathway for movement? The system should have a clear structure so directions for movement are obvious. Other ways of remedying problems at this step are to give overview maps of the VE or controls to change the viewpoint. (ii) Can the user execute movement and navigation actions? Poor design of the self representation may cause problems at this step. The same suggestions apply as for question (iii) in section 3.2. (iii) Can the user recognise objects in the environment? Objects may be obscured or not clear. The design should represent objects in detail with a clear outline to enhance recognition. (iv) Can users interpret the identity, role and behaviour of objects? Interpretation depends on domain knowledge. User learning may be supported by making objects explain themselves when approached or upon manipulation. Interaction may be necessary for interpretation of role and behaviour. Important objects should be represented in detail so that they can be interacted with. Active objects should be cued. (v) Can users remember important objects or locations? Memorisation can be helped by designing important locations or objects in the environment to be salient, e.g. use of colour, movement, size or shape. Set important objects apart from others; make key objects stand out as oddities among a set. Other memory support facilities are waymarks or tags (GDP 8) that the user can place in the VE, a visit list so users may inspect where they have been, and
6
replayable traces of previous explorations. The outcome of exploratory browsing should be improved user knowledge of the VE and external memory tags to help future exploration and task action. 3.4 SYSTEM INITIATIVE In this cycle the user responds to system behaviour until either initiative is relinquished by the system or the user interrupts system action. There are two variants of system initiative. Either the system may take the complete initiative, in which case the user has little choice but to be passive, e.g. in a guided tour; or agents within the system exhibit behaviour, although this does not preclude user action. In the latter case concurrent system and user actions are possible. The walkthrough questions, GDPs and use knowledge for the system initiative model are given in table 3. Table 3. Action in the system initiative model cross referenced to walkthrough questions, GDPs and user knowledge. (i) Is it clear to the user that the system has taken control? The onset of system initiative should be clearly signaled to the user (GDP1), and appropriate conventions adopted in conversational applications, e.g. gesture by mannequins when they wish to speak or act. (ii) Are the effects of system actions visible and recognisable? Care should be taken that actions in a remote part of the VE are made visible to the user or the presence of the active agent is cued. System actions should be easy to identify (GDP2). (iii) Are system actions interpretable? This depends on actions being known to the user. Actions and gestures carried out by agents should conform to conventions expected by the user. Actions should generally conform to the laws of physics unless such deviations are explicitly signaled. (iv) Can the user decide what to do next? The VE should provide affordances for user action (GDP 3) and signal when these actions can be taken (GDP 5). Use actions should normally always be valid so the user can regain the initiative at any time. (v) Is the end of system action clear? The end point of system initiative should be clearly signaled so the user knows when to resume command (GDP 5). In conversation applications, signalling of turn taking is particularly important. This may be achieved by speech discourse markers, by gesture by avatars or by explicit messages. (vi) Can the user resume control and is the appropriate action clear? In general, initiative should always reside with users so they are not frozen out of the application. The self should remain active when system agents are exhibiting behaviour as the user may wish to ignore them.
The same questions are used for both complete and partial system initiative, although in partial system initiative, signalling conventions may be more complex. For instance, it may be important to signal when concurrent activity is acceptable, and when the system agent has important information to convey as opposed to taking context setting actions. Ultimately there are many further complex questions in the realm of social control of virtual collaborative environments which we can not address in this method. BUSINESS PARK EXAMPLE Space precludes an exhaustive description of all the evaluation tasks in this paper, so the evaluation is illustrated with three representative tasks: Task (a) Finding out about equipment in the main building. This entailed discovering the functionality of the equipment as well as its location. This task implies goal directed exploration, so questions from the task cycle are used. No problems were apparent for walkthrough stages (i) to (iii) (see figure 5) in finding out about the office equipment. Stage (iv) gave problems as it was difficult to approach and orient towards the draftsman’s table which was at a slight angle; as depicted in figure 5. Deciding what action to take (stage (v)) was even worse as the object gave few cues about how it might be manipulated. A handle did suggest action and was active, but the board did not move until the handle had been pulled back. Stage (vi) was
7
blocked by the inability to find the correct cues, but when they were discovered manipulation was not easy. The effect of action was visible and interpretable as the drawing table changed orientation. Figure 5. Usability problem with approaching and orienting to the object when deciding how to operate the table in the task action model. It is not clear how the table controls can be accessed. Another example, illustrated in figure 6, was testing the electrical switch gear. Stages (i)-(iv) gave no problems, but stages (vii) and (viii) were not supported as the feedback (a small red light) was not easily visible and was difficult to interpret. Figure 6. Poor feedback problem after action had been taken to operate electrical switches. The red ‘on’ light was barely visible in the virtual environment. The i symbols indicate presence of additional information which can be accessed to describe the object – after application of GDPs 5 & 7 recognisable objects and clear object components.
The walkthrough indicated that design changes were necessary to make the objects interact more faithfully according to their real world model, and to provide better affordances and feedback from action. Improving the self-presence may also help alleviate manipulation problems. Task (b) Exploring the layout of the building, including the layout of rooms within it and their function. In the exploration analysis we focus on the toilets as these presented particular problems for orientation. The task was to find out what facilities were present in the toilets, and whether they were equipped for the disabled. This was goal directed exploration. Following the exploration cycle in figure 6, stages (i) and (ii) presented some problems as it was not clear where the toilets were located. To some extent this is a limitation of not knowing the environment. However, the real problems start with stages (iii) and (iv). The rooms, depicted in figure 6, are small and sparsely detailed. It is easy to get disoriented by an occluded view if the viewpoint is moved too close to a wall. It is difficult to examine the facilities as the viewpoint can not be set sufficiently far away for an overview without going backwards through a wall. Furthermore, several objects are inactive (e.g. water taps in basin), so they do not provide affordances for action thereby leading to misinformation about the real world that the VR was supposed to represent, i.e. plumbing facilities in the building. Better details in the rooms and more functional objects might help. Also an overview map of the building floor plan could help locate the toilets and improve user orientation about which room they were actually in. Task (c) Exploring the general layout of the Business Park, the access roads and its buildings. This is partially helped by a system initiated guided tour which starts on entry to the VE as illustrated in figure 4. In this case the system initiated walkthrough is used. It was not apparent that the system had taken the initiative (stage (i)). The user’s view in the VE is automatically moved on the guided tour but no explanation was provided and no cue of the mode change was given. It was difficult to resume control as the escape command implemented on a function key was not visible to the user (offending GDP 3, in stage (ii)), and the effects of the system’s actions were only partially recognisable and interpretable (stages (iii) and (iv)). These problems were compounded if the user tried to operate the 3D mouse unaware of the system’s initiative. Finally the end point of the tour (stage (v)) was cryptically signalled by the end of movement when the viewpoint was placed back at the starting point of the drive. The walkthrough suggests several design changes to make the initiative mode clear to the users and to allow resumption of user initiative at any moment. In conclusion the walkthrough uncovered several problems in the Business Park application ranging from poor affordances for interaction to poor navigation support and inappropriate system initiative. The majority of problems related to poor affordances for action and inadequate feedback when objects in the VE didn’t act as expected.
4. METHOD EVALUATION The GDP guidelines and method were evaluated by VR designers who were not HCI experts. The designers were given a tutorial and then carried out case study evaluation by walking-through products to test whether the method’s advice was comprehensible and effective. They were asked to rate the method and guidelines on a 1 to 5 scale for comprehensibility, helpfulness in diagnosing usability
8
problems, and utility in suggesting cures for usability problems. The overall ratings were favourable (averages 3.7 to 4.2). The method was reasonably easy to understand, although the designers did not always understand the difference between the explore navigate and task cycles. The diagnostic questions were followed without problems and uncovered several errors. In some cases the designers felt that usability errors might not be identifiable because the naturalness of the VE was more important than explicit user guidance. The usability errors identified by the method walkthrough were also compared with observed usability problems using cooperative evaluation method (Monk et al, 1993) with the Business Park application and 16 users. The walkthrough method correctly identified 80.4 % of the observed problems; furthermore, most of the non-identified problems occurred infrequently (1 or 2 times in 351 observed critical incidents and breakdowns). The more common problems missed by the method (> 5 incidents) were users not being able to locate objects or actions in the VE, and navigation difficulties caused by poor understanding of the spatial layout of the VE. The evaluation results are reported in more depth in Kaur et al (1999) and Kaur (1998).
5. DISCUSSION We have expanded previous cognitive walkthrough methods (Wharton et al, 1994) by proposing extensions of Norman’s (1986) model to deal with display based VR interaction. The method was easy to use and did uncover several errors; however, further testing is necessary to establish its effectiveness with other users and compare its analytic power against other techniques, e.g. Nielsen’s (1993) heuristic evaluation. In our future work we will extend the method to cover diagnostic evaluation thereby providing guidance that links observation of users’ problems to a diagnosis of problems and the design cures for VEs. The basis for this has already been laid in validation studies on the theory that compared observed usability problems against problems predicted by the GDPs and theory at specific stages in interaction (Kaur 1998). Over 80% of the observed usability errors were predicted by the theory. Although walkthrough methods are a good starting point for assessing VR usability, they offer no panacea for a thorough analysis. Studies of other usability inspection methods (Nielsen, 1993) have shown that trained personnel discover more problems and are better at prioritising their severity; however our users who were not HCI experts did predict 80% of the observed usability problems. While walkthrough methods may be effective in pointing out problems, they are less helpful on design solutions. We have countered that problem by attaching design advice to walkthrough questions and providing heuristics which can be used as design principles. The generic design properties may be extended to more specific design advice. We have made a start in this direction (Kaur et al, 1999), although the problem of context will be hard to solve. Some GDP guidelines may be inapplicable because the natural representation of the real world prevents their implementation, e.g. VR for marine navigation (Darken & Sibert, 1996) and we have not investigated collaborative VEs (see Benford et al 1995). A key question in VR applications is the naturalness of interaction, which may militate against ease of use as pointed out by Fischer (1993). Johnson (1998) also points out the need for task fit, natural representation and ease of navigation in a set of design principles for VR. Difficult environments may be a consequence of the task; alternatively, complex worlds may promote active learning so a trade off has to be established between constraints on natural representation and usability. Interpretation of problems will always require product and domain knowledge; however, our method can guide error diagnosis by use of simple question checklists thus, we believe, providing effective advice for non-HCI expert evaluators.
ACKNOWLEDGMENTS We wish to thank VR Solutions and The Rural Wales Development Board for loan of the application used in the study. In particular, we thank Phillip Trotter, Bob Stone and Andrew Connell. Kulwinder Kaur is supported by EPSRC post-graduate studentship 94314576.
9
REFERENCES BENFORD, S., GREENHALGH, C., BOWERS, J., SNOWDON, D. & FAHLEN, L.E. (1995). User embodiment in collaborative virtual environments. In CHI’95 Proceedings (pp. 242-249). New York: Association for Computing Machinery. BOLAS, M. (1994). Designing virtual environments. In C.G. Loeffler & T. Anderson (Eds) The Virtual Reality casebook. New York: Van Nostrand Reinhold. DARKEN R.P. & SIBERT J.L. (1996). Wayfinding strategies and behaviours in large virtual worlds. In: CHI '96, Vancouver 1996. Proceedings. (pp. 142-149). New York: Association for Computing Machinery. FISCHER, G. (1993). Beyond human computer interaction: designing useful and usable computational environments. In J.L. Alty (Ed) People and Computers VIII, Proceedings of HCI’93. (pp. 17-31). Cambridge University Press. HOOK, K. & DAHLBACK, N. (1992). Can cognitive science contribute to the design of VR applications? Paper presented at 5th MultiG Workshop, Stockholm, December 1992. ftp://ftp.kth.se/pub/MultiG/conferences/MultiG5. JOHNSON, C. (1998). On the problems of validating desktop VR. In People and Computer XIII, Johnson H., Nigay L. And Roast, C.(Eds) (pp 327-338), Berlin: Springer-Verlag. KAUR, K. (1998). Designing virtual environments for usability. Doctoral Thesis, Centre for HCI Design, School of Informatics, City University. KAUR, K., MAIDEN, N.A.M. & SUTCLIFFE, A.G. (1996). Design practice and usability problems with virtual environments. In: Virtual Reality World '96 conference, Stuttgart, Proceedings (IDG Conferences). KAUR, K., MAIDEN, NA.M. & SUTCLIFFE, A.G. (1999) Interacting with virtual environments: an evaluation of a model of interaction, Interacting with Computers, Vol 11, pp 403-426. KITAJIMA, M. & POLSON P.G. (1997), A comprehension based model of exploration. Human Computer Interaction, 12 (4), pp 345-390. LEWIS, C., POLSON, P., WHARTON, C. & REIMAN, J. (1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In J. R. Chew & J. Whiteside (Eds) Proceedings CHI-90., (pp. 235-241). New York: ACM Press. LOEFFLER, C.E. & ANDERSON,T. (1994). The virtual reality casebook. New York: Van Nostrand Reinhold. MILLER, L.D. (1994). A usability evaluation of the Rolls-Royce virtual reality for aero engine maintenance system. Masters Thesis. University College London. MONK, A., WRIGHT, P., HABER, J. & DAVENPORT, L. (1993). Improving your human computer interface. Prentice Hall. NIELSEN, J. (1993). Usability engineering. New York: Academic Press. NORMAN, D. A. (1986). Cognitive engineering. In D. Norman and S. Draper (Eds), User centred system design: new perspectives on human computer interaction, (pp. 31-62). Hillsdale NJ: Lawrence Erlbaum Associates. REIMAN J. & YOUNG R.M. (1996). A dual-space model for iteratively deepening exploratory learning. International Journal of Human-Computer Studies (44) 743-775. SUCHMAN L.A. (1987). Plans and situated actions: the problem of human-machine communication. Cambridge: Cambridge University Press. SUTCLIFFE, A.G., RYAN M., SPRINGETT M.V. & DOUBLEDAY, A., (in press). Model mismatch analysis: towards a deeper explanation of users’ usability problems. To appear in Behaviour and Information Technology. WANN, J. & MON-WILLIAMS, M. (1996). What does virtual reality NEED?: human factors issues in the design of three-dimensional computer environments. International Journal of HumanComputer Studies, 44, 829-847. WHARTON, C., REIMAN, J., LEWIS, C. & POLSON, P. (1994). The cognitive walkthrough method: a practitioner’s guide. In J. Nielsen & R.L. Mack (Eds) Usability inspection methods, (pp. 105140). New York: J Wiley.
10
11
Figure 1. VR interaction model for the task action cycle (after Norman, 1986). The nodes describe the user’s cognitive actions and the arcs show the possible pathways between actions.
Locate active 2 environment
objects not visible
Task goals 1 Form Goal
appropriate env. located intention formed
3 Locate objects
end of procedure
objects for action
8 Interpret Feedback satisfactory action
action selected
9 Specify next action
4 Approach & Orient
changes
action selected
7 Recognise Feedback
self positioned for action
object & env changes
effects
To explore cycle
6 Manipulate Object
5 Specify action action details
12
Figure 2. Interaction model for user behaviour in response to system initiative (i.e. action by an avatar in a virtual environment).
1 Acknowledge system control
Complete system initiative
user passive
2 Recognise change
5 Take responsive action To task regain action initiative
Partial system initiative
remain passive
4 Decide response
perceived system action
understood system action interrupt opportunity for action
13
3 Interpret system action
Figure 3. Interaction model for user exploration and navigation in virtual environments. During interaction the user’s behaviour will interleave this model and figure 1 as bouts of exploration and goal direction action occur.
1 Find path to target
Search target
5 Record/ memorise object interesting objects Target located
pathway identified
No search target
2 Decide direction continue exploration recognised location 4 Interpret change
change in location
14
movement selected 3 Move/ navigate
Figure 4. VR application for explaining facilities in a business park, courtesy of the Welsh Rural development agency.
15
Figure 5. Usability problem with approaching and orienting to the object when deciding how to operate the table in the task action model. It is not clear how the table controls can be accessed.
16
Figure 6. Poor feedback problem after action had been taken to operate electrical switches. The red ‘on’ light was barely visible in the virtual environment. The i symbols indicate presence of additional information which can be accessed to describe the object – after application of GDPs 5 & 7 recognisable objects and clear object components.
17
Action step 1. Form goal
2. Locate active environment
3. Locate objects
Question (i) Can the user form or remember the task goal ? (ii) Can the user form an intention of what to do ? (iii) Are the appropriate objects or part of the environment visible ? (iv) Can the necessary objects be located ?
4. Approach and orient
(v) Can the users approach and orient themselves to carry out the necessary action ?
5. Specify action
(vi) Can the user decide what action to take and how ? (vii) Can the user carry out the manipulation easily?
6. Manipulate object
7. Recognise Feedback 8. Assess Feedback
9. Specify next action
(viii) Is the consequence of action visible? (ix) Can the user interpret the change? (x) Can the user decide what to do next?
GDPs 1. Compatible task flow
User knowledge Task knowledge Domain knowledgeappropriate situation
2. Clear environment structure
Domain knowledge- env layout
2. Clear environment structure. 5. Recognisable objects 6. Approachable object 7. Clear object components 4. Flexible self operation 3. Affordance for action
Domain knowledgeobjects and location
4. Flexible self operation 7. Clear object components 8. Locatable areas for manipulation 9. Visible effect of change 10. Interpretable effect
3. Affordance for action
Domain knowledge- object structure, Task knowledgeorientation actions Task knowledgeaction detail
Domain knowledgeexpected effects Domain knowledge, task knowledgeexpected effects Task knowledge
Table 1. Actions in the task cycle cross reference to walkthrough questions, GDPs and user knowledge for successful interaction.
18
Action
Question
GDP
User Knowledge
1. Find path to target
(i) Does the user know where to start looking ?
1.
Clear environment structure. Observable pathways Affordance for navigation, Clear object components Flexible self operation. Recognisable objects
Domain knowledge- VE layout
Recognisable objects Visible effects of movement Waymarking, trace pathway facilities
Domain knowledgeobjects, VE structure
2. 2. Decide direction
3. Move Navigate
4. Interpret change
5. Record / Memorise location
(ii) Can the user determine a pathway towards the target ? (iii) Can the user execute movement and navigation actions? (iv) Can the user recognise the search target ?
(v) Do facilities for recording locations exist?
3.
6 4. 5.
6. 7.
8.
System knowledgenavigation actions
System knowledgeNavigation actions
---
Table 2. Actions in the exploration/navigate cycle cross referenced to walkthrough questions, GDPs and user knowledge.
19
Action 1. Acknowledge system control 2. Recognise change
3. Interpret system action 4. Decide response
5. Take responsive action
Walkthrough question (i) Is it clear to the usr that the system has taken control? (ii) Are the effects of system actions visible and recognizable ? (iii) Are system actions easy to interpret (iv) Can the user decide what to do next? (v) Is the end of system action clear? (vi) Can the user resume control?
GDP 1. Initiative status change 5. Initiative turn signal 2. Observable change
User knowledge Domain- agent behaviour. System controls Domain knowledge
4. Interpretable action
Domain knowledgeagent behaviour Domain and task knowledge
3.
Affordance for user initiative 5. Initiative turn signals 6. Clear initiative control command
Task knowledge
Table 3. Action in the system initiative model cross referenced to walkthrough questions, GDPs and user knowledge.
20
