Qualitative Usability Measurement of Websites by Employing the Repertory Grid Technique Jouke C. Verlinden Faculty of Design, Engineering and Production Delft University of Technology Jaffalaan 9 2628 BX Delft, the Netherlands +31 15 2789321
[email protected] ABSTRACT
In this paper, we describe some initial work on measuring the subjective or qualitative usability aspects of websites. Our main motivation is to facilitate the human ability to compare parts of a website. Keywords
Usability measurement, website evaluation, semantic mapping INTRODUCTION
The contribution of design disciplines to the realization of usable websites seems indisputable. However, so called designed websites still show shortcomings with respect to appreciation, accessibility, informative content. In this short talk, we want to discuss an additional method for usability research whose primary focus is the comparison (by users) of parts of a website. THE CURRENT PROBLEM
Even when user-centered design is part of the design process, the end result is often not evaluated. A wide range of performance and usability measurement methods exist. Only a few support the mapping of how human beings perceive and value a site. Yet a better understanding of the subjective experience of end users will be very important in dealing with issues such as: receive attention of users, leveraging valued parts of a website, bring about emotional well being. Most usability techniques are quantitative and measure the performance, retrieval times, success times/failure rates. Qualitative assessments usually consist of observations and interviews. Although this might provide some information on experiences with respect to websites, it exposes a number of shortcomings:
Marc J.J. Coenders Informaat BV Jacob van Lenneplaan 57 3740 AT Baarn, the Netherlands +31 35 543-1222
[email protected] • The results of open questions in assessments are hard to evaluate and aggregate. Same comments can have different meanings. This makes it difficult to compare the results of observations and interviews. • With standardized questions, the designer’s view is imposed on the respondent, which could focus on the wrong themes. • Observations and open interview deliver global conclusions. What does the expression “The website is bad” mean? Should the whole site change or only a small part? • It does not exploit one of the most powerful capabilities humans have: to compare parts. That is the relative measurement of things as opposed to an abstract, absolute rating. The purpose of the proposed method is to fill the gap between general knowledge about usability (guidelines, standards, theories) and knowledge of users (human factor specification). In our method, users evaluate a website internally. METHOD
Our method is loosely linked with the personal construct theory and repertory grid (Reger, R.K.) and theories about perception and semantic distance (Rips, Lance J.). The repertory grid technique was developed by George Kelly on the basis of the Personal Construct Theory. One of the goals was to exclude the researcher’s frame of reference and worldview. In short a typical repertory grid session consists of the following steps: Step 1: Eliciting constructs. Each construct consists of a likeness and contrast pole (we refer to these as ‘dimensions’). Step 2: Indexing elements on these constructs. Step 3: Quantification of the perceptions of end users through rating, ranking and/or sorting of elements. Step 4: Statistical analysis to find out about element distance, construct centrality, and preferences.
Considering the problems stated earlier, the potential of this technique is immense. It facilitates an objective approach of capturing subjective aspects of web usability. Furthermore, the technique can be easily applied to a large test group with the use of on-line tools.
CONCLUSIONS AND FURTHER RESEARCH
Another important aspect is that the results of the tests are easily mapped into diagrams, one of the best ways to communicate user experiences to all stakeholders in the development of websites.
Contributions
APPLICATION
The first steps towards translating this technique towards the field of user interface design were quite obvious. We applied this during the evaluation of a website of a large insurance company based in the Netherlands. Directly after a traditional success time/failure rate test, the respondents were asked to compare a selection of web pages and to map those into a grid. This resulted in diagrams like: attractive •
car insurance
• •
financial services • entertainment policy information
difficult •
easy news
• homepage • company info. irritating
In some other sessions, the respondent had to rank the selection of pages separately for each dimension. This might be a better way to rate the items in more than two dimensions. For sake of saving time, we sped up the first two steps by providing the respondents a pre-selection of three dimensions, from each respondent had to select two: 1.
feel (attractive versus irritating)
2.
use (easy versus difficult)
3.
informative (interesting versus uninteresting)
The results of these measurements could be easily combined into composite diagrams, which acted as maps that captured the experiences of a representative user group. These maps could be employed to determine the most prominent areas to improve. They also indicate which clusters of information weren’t considered important. These first trials were performed with pen-and-paper. However, we are in the process of creating web-based tools to perform these kinds of evaluations on-line.
The technique that we used to capture subjective experiences is in its inceptive stages. However, the evolved version could be added to the testing palette of usercentered design and usability engineering methods. Based on the current insights, we think the method can offer the following contributions: • Users hold certain mental models about what is pleasant, beautiful or harmonious. This technique can become an important tool to capture those constructs. • It can help designers to draw the design and solution space for usability issues in the sense that possibilities and limitations become clear. If users value text and pictures pages more than text and video, it is obvious that more video does not make sense. • It elucidates certain user characteristics that are not present in a human factor specification. In fact, the research could lead to another classification of users. Shortcomings
• Comparison among several parts of the same website might not elicit a great number of differences. • In contrast to clustering analyses or task analysis, this technique does not provide information on the paths that users would like to follow on the website. Each part is valued separately. • Because the technique uses only existing parts of a website, it cannot be used to capture needs and expectations that are not present in the current version. Future work
Based on the current insights, this technique needs to mature before becoming widely applicable. • The interviewing protocol should be expanded to encompass online testing. It will be challenging to determine what part of the test can be performed without having to go to a lab or to meet an interviewer. • In the future, the discovery of the dimensions those during the sessions themselves or at least in a preliminary test group. REFERENCES
1. Reger, R.K. The Repertory Grid Technique for Eliciting the Content and Structure of Cognitive Constructive Systems. Mapping Strategic Thought. John Wiley&Sons Ltd, 301-309, 1990. 2. Rips, L.J., Shoben, E.J., et Smith, E.E. Semantic distance and the verification of semantic relations. Journal of Verbal Learning and Verbal Behavior, 12(1973), 1-20.
