analysis of the Health Information Technology Usability Evaluation Scale (Health- ..... but also by the degree to which it can be successfully integrated to perform ...
Health Information Technology Usability Evaluation: Methods, Models, and Measures
Po-Yin Yen
Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy under the Executive Committee of the Graduate School of Arts and Sciences
COLUMBIA UNIVERSITY 2010
UMI Number: 3420882
All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion.
UMT Dissertation Publishing
UMI 3420882 Copyright 2010 by ProQuest LLC. All rights reserved. This edition of the work is protected against unauthorized copying under Title 17, United States Code.
ProQuest LLC 789 East Eisenhower Parkway
P.O. Box 1346 Ann Arbor, Ml 48106-1346
©2010 Po-Yin Yen All Rights Reserved
ABSTRACT Health Information Technology Usability Evaluation: Methods, Models, and Measures Po-Yin Yen Health information technology (IT) can offer important benefits to health care; however, technology-related factors are a major obstacle to health IT adoption. Toward the goal of achieving a greater understanding of health IT usability and its measurement, the dissertation comprised three major analyses: 1) a methodological review of health IT usability evaluation studies to identify problems in existing studies; 2) exploratory factor analysis of the Health Information Technology Usability Evaluation Scale (HealthITUES) which was developed as part of the dissertation research along with the underlying Health Information Technology Usability Evaluation Model (Health-ITUEM); and 3) confirmatory factor analysis and structural equation modeling to examine the construct validity and predictive validity of Health-ITUES. The health IT system that served as the focus of the analysis was a web-based communication system that supported nurse staffing and scheduling. The sample comprised 553 staff nurses in two healthcare organizations. In the usability methodological review, we identified problems in existing studies including lack of theoretical framework/model, inconsistent usability definition and evaluation methods, and lack of power analysis for sample size calculation. The exploratory factor analysis resulted in a 20-item Health-ITUES comprising four factors that demonstrated strong internal consistency reliability: quality of work life (QWL), 3
items, a=.94; perceived usefulness (PU), 9 items, a=.94; perceived ease of use (PEU), 5 items, a=.95; user control (UC), 3 items, a=.81. The confirmatory factor analysis showed that a general usability factor accounted for 78.1%, 93.4%, 51.0% and 39.9% of the explained variance in QWL, PU, PEU, and UC respectively. The structural equation modeling supported the predictive validity of Health-ITUES, explaining 64% of the variance in intention for system use. The results of the dissertation contribute to enhancing the methodological breadth and rigor of health IT usability evaluation studies.
TABLE OF CONTENTS Chapter 1. Introduction
1
Background
2
Definitions of Usability
3
Aspects of Usability
7
Definition and Scope
8
Health IT Usability, Acceptance, and Adoption
10
Health IT Usability Specification and Evaluation
11
Problem Statement
12
Purpose
13
Study Aims and Research Questions
13
Significance of the Study
15
Chapter 2. Methodological Review of Health Information Technology Usability Specification and Evaluation Studies Background
16 16
Usability Model
17
System Development Life Cycle
17
An Integrated Usability Specification and Evaluation Framework
18
Methods
23
Search Strategy
23
Inclusion/Exclusion Criteria:
23
Data Extraction and Management
,
Results
24 27
Types of Health IT Evaluated
32
Summary of studies categorized in each stage i
33
Stage 1: Specify Needs and Setting
35
Stage 2: System Component Development
37
Stage 3: Combination Components
38
Stage 4: Integrate Health IT into the Real Environment
40
Stage 5: Routine Use
41
Study Design and Data Analysis in Stages 4 and 5
41
Discussion
45
Methodological Problems in Existing Studies
45
Objective versus Subjective Measures
47
Environmental Factor Not Evaluated in the Early Stages
49
Inadequate Measure
49
Limitations
50
Conclusion
50
Chapter 3. Health IT Usability Evaluation Model and Scale Development
65
Background
65
Health IT Usability Evaluation Model
70
Definition of Concepts
74
Health IT Usability Evaluation Scale Development
76
The web-based communication system
76
Item selection, creation, and modification
78
Health IT Usability Evaluation Scale Psychometric Evaluation
82
Research Questions
82
Methods
82
Results
84 ii
Discussion
92
Limitations
94
Conclusion
95
Chapter 4. Health-IT Usability Evaluation Scale Confirmatory Analyses
96
Background
96
Research Questions
96
Methods
97
Setting and Sample
97
Sample size
97
Data collection procedures
98
Data Analysis
98
Results
99
Descriptive analysis
99
Power Analysis
100
Construct validity
101
Predictive Validity
104
Discussion
106
Construct and Predictive Validity
106
Methodological issues in existing model testing studies
106
Limitations
107
Conclusions
108
Chapter 5. Discussion
109
Study Findings
110
A Stratified View of Health IT Evaluation
Ill
iii
Guidance for Health IT Evaluation
113
Health-ITUEM and Health-ITUES
117
Implications for Nursing and Nursing Informatics
120
Future Research Directions
121
Improving the literature review process through social network analysis
121
Usability evaluation methodological improvement
122
Moving from cross-sectional study design to longitudinal study design to improve assessment of directionality and causality
123
Model comparison for competing theories
124
Conclusion
125
References
126
IV
LIST OF TABLES Table 1-1. Definitions of usability
5
Table 1-2. Usability Aspects
7
Table 1-3. Study Aims and Research Questions
14
Table 2-1. Usability Specification and Evaluation Framework
20
Table 2-2. Data Extraction
24
Table 2-3. Criteria for Study Categorization Based on the SDLC Stages
27
Table 2-4. Search Strategies and Results
28
Table 2-5. Number of health ITs being evaluated based on health IT type
32
Table 2-6. Methods by evaluation type
33
Table 2-7. MeSH terms for Stage 2 study identification
37
Table 2-8. Study design and data analysis
43
Table 2-9. Theory/Model/Framework
52
Table 2-10. Methods for Usability Specification and Evaluation
59
Table 3-1. Questionnaires
67
Table 3-2. Identification of Health-ITUEM Concepts..
74
Table 3-3. Definitions of Health-ITUEM Concepts
75
Table 3-4. Health-ITUESItems Mapped to Health-ITUEM
79
Table 3-5. Internet Competency of Staff Nurses
85
Table 3-6. Communalities
86
Table 3-7. Factor loadings
87
Table 3-8. Factor correlations
88
Table 3-9. Reliability of Each Factor before Item Deletion
89
Table 3-10. Factor Loadings after Item Deletion
90
v
Table 3-11. Reliability of Each Factor after Item Deletions
90
Table 3-12. Factor Correlation after Item Deletions
91
Table 3-13. Health-IT Usability Evaluation Scale (Health-ITUES)
91
Table 4-1. Demographics of CUMC-NYP Staff Nurse Participants (n=176)
99
Table 4-2. CFA model fit indices
104
Table 4-3. SEM model fit indices
105
Table 5-1. Guide for Selection of Theories, Outcomes, and Methods Based on Type of Interaction
115
vi
LIST OF FIGURES Figure 1-1. Usability Definition (ISO 9241 vs. ISO 9126)
4
Figure 1-2. Bennett's model of usability
6
Figure 1-3. Interaction Components and Aspects for Technology Usability Evaluation... 9 Figure 1-4. Scope of the Dissertation
9
Figure 1-5. Health IT Usability, Acceptance, and Adoption
11
Figure 2-1. Data management flowchart
32
Figure 3-1. Health IT Usability Evaluation Model (Health-ITUEM)
73
Figure 3-2. The web-based scheduling and staffing system
77
Figure 4-1. Power analysis
100
Figure 4-2. CFA 1st order model
102
Figure 4-3. CFA 2nd order model
103
Figure 4-4. Structural equation model to predict intention to use
105
Figure 5-1. A Stratified View of Health IT Evaluation
113
Figure 5-2. Health-ITUEM
119
Figure 5-3. Social Network Analysis implication for literature review process
122
Figure 5-4. Longitudinal study plan for usability evaluation
124
Figure 5-5. Second order model vs. bi-factor model
125
vii
ACKNOWLEDGEMENT
I have many people to thank for their support and advice given throughout this process. My deepest gratitude goes to my advisor, Dr. Suzanne Bakken, for her generous support in every way throughout my doctoral journey. She gave me the freedom needed to explore my own interests while providing me with the necessary guidance when I was lost. I always felt amazingly fortunate for being her student. Her generosity, love and positive influence taught me not only in school, but in every area of my life. I also want to thank my committee members, Dr. Karen Sousa, Dr. Haomiao Jia, Dr. David Kaufman, and Dr. Patricia Stone, for their expertise and guidance. Dr. Karen Sousa, even though far away in Colorado, was generous with her time and knowledge. Dr. Haomiao Jia constantly provided me with novel perspectives. Dr. David Kaufman and Dr. Patricia Stone provided great feedback that stimulated my thinking. It was a wonderful experience working with my committee members and I sincerely appreciated their help throughout this process. I want to extend my thanks to the individuals who helped me with my dissertation research. Douglas Hughes and Catherine DiNardo from Main Line Health were instrumental to my data collection. The assistance provided by Georgia Persky and Grace Su from New York Presbyterian Hospital helped in the forward movement of my research project. My sincere gratitude to all of the nursing professions who participated in my study. I also wish to thank my colleagues in the "Pod". I enjoyed my graduate student life with them, both in class and in social life. Their support and care helped me viii
overcome the rigors of the dissertation process. Thank you very much. Additional thanks goes to my family and friends for their unfailing love and encouragement. They have always been an important and indispensable source of my spiritual support. The dissertation was supported by the Center for Evidence-based Practice in the Underserved (P30NR010677).
IX
DEDICATION To my dearest family
x
1
Chapter 1.
Introduction
Health information technology (IT) is defined as "the application of information processing involving both computer hardware and software that deals with the storage, retrieval, sharing, and use of health care information, data, and knowledge for communication and decision making" (Thompson, T. & Brailer, 2004). Applications are systems used by clinicians or patients, such as electronic health records, computerized provider order entry (CPOE), clinical decision support systems (CDSSs), nursing information systems, electronic prescribing systems, personal health records, and telemedicine. Health IT offers important benefits to healthcare, including decision support, knowledge management, improved communication, effective resource management, and reduction of medical errors. Additionally, it can save time and reduces paperwork (Shortliffe, Perreault, Wiederhold, & Fagan, 2001). However, even though these benefits have been recognized, studies show that only 9.6% of hospitals have CPOE (Ash, Joan S., Gorman, Seshadri, & Hersh, 2004), and electronic health records are used in only 18% of all ambulatory visits (Linder, J. A., Ma, Bates, Middleton, & Stafford, 2007). The Healthcare Information and Management Systems Society (HIMSS) analyzed electronic medical record (EMR)1 adoption rates using the EMR adoption model, an 8-stage scale that scores hospitals from Stage-0, no information system installed, to Stage-7, a fully paperless environment. The results indicated that 90% of U.S. hospitals are in Stage-0 to Stage-3.
Healthcare Information and Management Systems Society (HIMSS) defines electronic health record (EHR) as a longitudinal electronic record of patient health information generated by one or more encounters in any care delivery setting, while electronic medical record (EMR) is a component of an electronic health record that is owned by the health care provider.
2 These stages only include major ancillary clinical systems (i.e., pharmacy, laboratory, radiology), clinical data repository, nursing/clinical documentation (e.g., vital signs, flow sheets), nursing notes, care plan charting, and electronic medication administration record systems. In addition, HIMSS reported that CDSSs are used mostly for error checking and order entry, and that other integrated functions of electronic health records have not been implemented (HIMSS Analytics, 2009). Usability factors pose one of the major obstacles to health IT adoption and include ease of use, usefulness, flexibility, relevancy, and completeness (Miller & Sim, 2004; Yusof, Stergioulas, & Zugic, 2007). The objective of this dissertation is to conduct a methodological review of usability studies and to develop and test an evaluation scale for health IT usability. Background Understanding the deficiencies in health IT is essential to predicting why a system is likely to fail. Current evaluation of health IT focuses more on the quality, efficiency, and cost of medical care. Improving these factors is the ultimate goal of health IT. However, studying them as outcomes can only identify the advantages and disadvantages of using health IT; it cannot inform health IT design for better usage. Usability evaluation is a method for identifying specific problems with the usability of products, and can be used to improve their usability (Dumas, J. S. & Redish, 1999; Yao & Gorman, 2000). The benefits of usability evaluation include improved predictability of products, greater productivity with fewer user errors, and savings in development time and cost (Dumas, J. S. & Redish, 1999; Nielsen, 1993; Wiklund, 1994).
3 Definitions of Usability The concept of usability was defined in the field of human computer interaction (HCI) as the relationship between humans and computers. The International Organization for Standardization (ISO) proposed two definitions of usability, ISO 9241 and ISO 9126. ISO 9241 defines usability as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" (ISO 9241-11, 1998). In ISO 9126, usability compliance is one of five product quality categories, in addition to understandability, learnability, operability, and attractiveness (ISO/IEC 9126, 2001). Usability depends on the interaction between user and task in a defined environment (Abran, Khelifi, Suryn, & Seffah, 2003; Bennett, 1984). Therefore, ISO 9126 defines usability as "the capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions" (ISO/IEC 9126, 2001). While this definition focuses on ease of use, ISO 9241 uses the term "quality in use" to describe usability more broadly (Abran, et al., 2003; Bevan, 2001) (Figure 1-1). "Quality in use" is defined as "the capability of the software product to enable specified users to achieve specified goals with effectiveness, productivity, safety, and satisfaction in specified contexts of use"(ISO/IEC 9126, 2001).
4
ISO 9241
ISO 9126
,, ; i : uaouuu^
.
T 1
"t*— ^
< n
C
sabilit = Eas of us
3
>
ttracti ness
^
03
O
perabi
*-*• >->!
r:
nderst dabil
o 3
t-1
earnab
ess
n> 3
c/o
atisfac
o
«
.70
Reliability
.71-.92
Reliability =
.59-.81
Reliability =
.88-.96
Reliability =
.91-.96
Reliability =
Reliability
validity
discriminant
Convergent and
Construct validity
Criterion validity
Content validity
Construct validity
Content validity
Construct validity
Content validity
Validity
70
Health IT Usability Evaluation Model The Health IT Usability Evaluation Model (Health-ITUEM) (Figure 3-1) is an integrated model that we developed based on multiple theories to inform a comprehensive view of usability evaluation. Definitions of usability from TAM and ISO 9241-11 provided the fundamental constructs of the Health-ITUEM, such as perceived usefulness and efficiency, while Bennett's model identified which objects need to be included. The Technology Acceptance Model (TAM) (Davis, Bagozzi, & Warshaw, 1989) is an extension of the Theory of Reasoned Action (TRA) (Ajzen & Fishbein, 1980), which proposed the predicted relationship between attitude and behavioral intention. TAM extended the TRA and suggests that perceived usefulness and perceived ease of use influence users' attitudes toward using the technology and contribute to their behavioral intention to use the technology. TAM has been broadly applied in healthcare to understand clinicians' perceptions of health IT and the evidence regarding its usefulness to predict clinicians' acceptance of new technology is substantial (Barker, et al., 2003; Chau & Hu, 2002; Chismar & Wiley-Patton, 2002; Day, Demiris, Oliver, Courtney, & Hensel, 2007; Hulse, Del Fiol, & Rocha, 2006). While TAM addresses subjective measures of usability, ISO 9241-11 encompasses objective measures of usability. ISO 9241-11 defines usability as: "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" (ISO 9241-11, 1998). Comprehensive usability evaluation should include both assessment of objective
71
and subjective measures (Hornbaek, 2006). We did not include two common concepts identified in the literature review, mandatory use and user variance, in Health-ITUEM because they are not direct measures of Health IT usability. Rather, they are measures of confounding factors that influence technology acceptance rather than usability. In the following paragraphs, we further describe our rationale. Venkatesh introduced mandatory use as a key influence on technology acceptance (Venkatesh & Davis, 2000; Venkatesh, et al., 2003). Several studies have examined technology acceptance in mandatory use environments. One study reported that lower perceived usefulness of technology in regards to valued outcomes, such as job performance is associated with negative attitudes, i.e., low satisfaction (Brown, Massey, Montoya-Weiss, & Burkman, 2002). Another study demonstrated that when users' initial expectations of the system are not met, users' perceptions of usefulness and ease of use were reduced (Sorebo & Eikebrokk, 2008). These studies provide evidence that level of expectation is key. If users perceive a system to be easy to use and useful to perform simple tasks, but do not perceive the system to be beneficial to valued outcomes or higher expectations, their satisfaction is low (Brown, et al., 2002). Thus, Health-ITUEM includes two concepts posited to directly represent higher level expectations, "Satisfaction" and "Other outcomes", rather than the indirect influence of mandatory use. User variances, such as subjective norms, age, gender, education and related experiences influence technology acceptance. For example, studies found that more educated users are better able to learn the benefits of new technologies (Agarwal and
72
Prasad, 1999), senior staff perceived the system to be more useful for their tasks than lower level staff (Burton-Jones & Hubona, 2005), and older workers reported lower perceived ease of use (Burton-Jones & Hubona, 2005). We do not include user variances in Health-ITUEM because it is a universal expectation and system features and user training are expected to facilitate variations in users. The impact of socio-organizational factors such as mandatory use and user variances on technology acceptance are well-acknowledged in the literature. However, their influence should be differentiated from health IT usability, in order to appropriately target improvement at health IT usability, user training or environmental supports.
Other outcomes
Information needs
Memorability
Learnability
Figure 3-1. Health IT Usability Evaluation Model (Health-ITUEM)
Error prevention
Perceived Ease of Use
Satisfaction
Subjective
Competency
Perform speed
Perceived Usefulness
Health IT Usability
Flexibility/ Customizability
Efficiency
Effectiveness
Objective
74
Definition of Concepts To further delineate each Health-ITUEM construct, perceived usefulness, perceived ease of use, effectiveness and efficiency, we identified concepts from usability decompositions (Folmer & Bosch, 2004; van Welie, van der Veer, & Eliens, 1999), Nielsen's ten heuristics (Nielsen, 2005), Shneiderman's eight rules for user interface design (Shneiderman, 1997), and Norman's seven principles for design (Norman, D., 1990). Discussion with potential system users and developers also recognized additional concepts. Table 3-2. Identification of Health-ITUEM Concepts Nielsen
Shneiderman
Norman
Folmer
van
IBM
User and
(1993)
(1998)
(1990)
(2004)
Welie
(1995)
developer
(1999) Safety/Errors
V
V
V
V
V
V
V
V
Completeness Memorability
V
V
V
V
V
Information needs Flexibility
V
V
V
V
Learnability
V
V
V
V
V
Performance
V
V
"o I—i 4^ t—» ON NO ON o ON LO o o ON
age
Lo to LO Lo t o to ^^ - J ON NO LO oo 4 ^ t o O N 4 ^ LA NO ON LA 0 0 ON to o 4 ^ LO
> t o to to t o to LA 4^ LO t o H-1 o NO GO o ON LA 4^ LO to •—' o
o p o O p o p o O p O p p p p p p Lo 4^ to LA Lo 4^ LA * • & . LA LA LA ON LA bo bo bo LA NO N> to t o H — 4*. 4 ^ 0 0 0 0 NO ON NO ~o 4i> oo - J o 0 0 ~ J ON LA 0 0 ON NO LA *"" NO *~ o LA oo LA o NO
1
t«3
TO
>3
& 3
