Review paper
What should we measure? Conceptualizing usage in health information exchange Joshua R Vest,1 ‘Jon Jasperson2 1
Department of Health Policy & Management, School of Rural Public Health, Texas A&M Health Science Center, College Station, Texas, USA 2 Department of Information and Operations Management, Mays Business School, Texas A&M University, Texas, USA Correspondence to Dr Joshua R Vest, Department of Health Policy & Management, School of Rural Public Health, Texas A&M Health Science Center 1266, TAMU, College Station, TX 77843, USA;
[email protected] Received 28 September 2009 Accepted 3 March 2010
ABSTRACT Under the provisions of the Health Information Technology for Economic & Clinical Health act providers need to demonstrate their ‘meaningful use’ of electronic health record systems’ health information exchange (HIE) capability. HIE usage is not a simple construct, but the choice of its measurement must attend to the users, context, and objectives of the system being examined. This review examined how usage is reported in the existing literature and also what conceptualizations of usage might best reflect the nature and objectives of HIE. While existing literature on HIE usage included a diverse set of measures, most were theoretically weak, did not attend to the interplay of measure, level of analysis and architectural strategy, and did not reflect how HIE usage affected the actual process of care. Attention to these issues will provide greater insight into the effects of previously inaccessible information on medical decision-making and the process of care.
INTRODUCTION Health information exchange (HIE) is the process of electronically moving health-related patient information between different organizations,1 2 and numerous policy makers, researchers, industry groups, and healthcare professionals contend it promises to improve the quality of the US healthcare system.2e5 Title XIII of the American Recovery & Reinvestment Act of 2009, also as known as the Health Information Technology for Economic & Clinical Health (HITECH) act dramatically advances the prospect for widespread HIE adoption by requiring providers who utilize electronic health records (EHRs) to have information exchange capability to be eligible for Medicare and Medicaid incentive payments.6 This incentive payment system, requiring demonstrated ‘meaningful use,’ significantly builds on the existing federal support for interoperable health information sharing begun under the previous administration.7 While this federal support will expand, or foster new exchange efforts, healthcare providers must actually use HIE systems in their work before any anticipated benefits can be recognized.8 9 Unfortunately, in many cases, the expected benefits of health information technology do not materialize owing to lack of use, misuse, or other forms of system rejection.10e18 The long-term, and potentially farreaching, effects of the HITECH act on the healthcare industry warrant a better understanding of what constitutes HIE usage. Those interested in HIE evaluation frameworks have accordingly prioritized understanding usage.5 19 20 The current policy implications of meaningful use and the context of historical difficulties with 302
health information technologies prompt the heretofore unanswered, question: what conceptualizations of usage might best reflect the nature and objectives of HIE? The existing literature on HIE should provide logical candidates for the measurement of HIE usage under meaningful usage requirements; however, these options may not be particularly informative or even appropriate in evaluating and improving efforts. Therefore, looking toward future opportunities for evaluation and research, this review aims to add to the current HIE evaluation dialog by (a) examining the concept of usage in relation to the nature and objectives of HIE, (b) reviewing the existing literature on HIE usage, and finally (c) identifying key considerations in selecting measures of usage specific to HIE in order to guide further research.
BACKGROUND HIE is a process that links and integrates an individual patient’s information from multiple, disparate, data sources.1 HIE may be accomplished through a variety of architectural strategies, which serve to define how organizations share, store, control, and access the information made available within the exchange relation. Depending on the architecture, the tangible outcome of the HIE process occurs by allowing end users to access another organization’s information system, using data to populate a novel information system available to users, or pushing/pulling data into existing electronic health records (EHRs).21 HIE is a process that (1) results in accessible information, (2) involves individual and collective actors, and (3) is influenced by the architectural strategy, each of which has a bearing on usage. An information system provides support to end user activities and functions by arranging and displaying information.22 23 As noted above, such information systems make the results of the HIE process accessible. Information systems researchers frequently conceptualize information systems use to include two different aspects: using the system and using the information provided by the system. First, users have access to the process of HIE through various cognitive artifacts, such as software and hardware. Employing the system features related to the performance of an organizationally defined task constitutes ‘using the system.’24 25 The second aspect of usage concerns the application of the information made accessible by the HIE process. DeLone and McLean conceptualized this type of usage as the ‘recipient consumption of the output of an information system.’26 Given ‘the primary purpose of health information exchange efforts today is to bring information concerning the
J Am Med Inform Assoc 2010;17:302e307. doi:10.1136/jamia.2009.000471
Review paper patient to the care delivery process,’27 we suggest both aspects of usage should be considered when examining meaningful use of HIE systems. Furthermore, information systems use occurs at various levels of analysis.9 28 For example, while an individual physician or nurse may be the prototypical end user when considering HIE, meaningful use may also occur as an activity of collectives of varying size, membership, and purpose.27 29 Above the level of the individual professional, teams or groups may employ information systems.30 In the US healthcare system, patients see multiple providers and receive care from groups of professionals representing different organizations or entities.31e33 Multiple individual professionals organized around a single patient as part of case management program or a medical home can use HIE as a collective entity. By definition, the HIE process includes exchange of information by organizations. Thus, the manner in which an organization applies and uses the HIE process has always been at the forefront of efforts to facilitate HIE and represents a third level of analysis.18 21 Lastly, the collaborative effort facilitating HIE, whether it is a regional health information organizations (RHIO) or some other interorganizational relationship, is the network level of analysis. Moreover, while an activity of organizations, the data exchanged within these networks are ultimately intended for utilization by individuals in the organization.27 Therefore, instead of merely distinguishing between the levels of usage, measures could be truly multilevel, reflecting the linkages and connections between and within levels like shared cognition.28 Third, the HIE architectural strategy will, in part, dictate allowable ways of measuring a health provider’s usage. For example, in any HIE effort that results in additional information systems for users to access, usage may be voluntary. In contrast, when multiple organizations utilize the same EHR to exchange information, no alternative information system exists and HIE usage essentially becomes mandatory. Alternatively, the choice of architectural strategy determines number of information systems available to users, When HIE is utilized to populate a novel information system, like a centralized data repository, it is in addition to the participating organizations’ own internal information system or EHR. Therefore users have two information systems to access. In contrast, when the architectural strategy allows users access to another organization’s information system, like a federated model, the number of available data repositories could be even greater. In this example usage may entail the employment of multiple different information systems. In an effort to understand how HIE usage has been studied and applied, we examined the existing literature for research involving HIE and the information exchange process. Our review was guided by the overarching goal to recognize (1) both the system employment and information consumption aspects of use described above, (2) different levels of analysis for use, and (3) the effect of the architectural strategy on HIE use studied.
REVIEW PROCESS We reviewed the literature for studies and reports measuring either the employment of HIE system features or the application of information obtained from an HIE. The following terms were searched in Medline and ISI Web of Knowledge: < ‘health information exchange’ < ‘electronic data interchange’ < ‘regional health information organization’ J Am Med Inform Assoc 2010;17:302e307. doi:10.1136/jamia.2009.000471
< ‘local health information organization’ < ‘local health information infrastructure’ < ‘community health information network.’
Terms were derived from the Healthcare Information and Management Systems Society’s definition and HIE acronym list.21 We limited results to peer-reviewed, English language, journal articles published between 1991 and 2008.i We also reviewed the journal articles cited in the recent Journal of Biomedical Informatics supplement on HIE evaluation for additional potentially relevant titles. This search strategy yielded 815 citations (the majority of which (n¼470), because of the broadness of the term ‘electronic data interchange’ were unrelated to healthcare). These 470 articles were excluded from further consideration. We reviewed the titles and abstracts of the remaining articles to determine which articles warranted a detailed full text evaluation (n¼80). From this set, we retained only those reporting HIE usage. We excluded articles that only discussed technological capabilities or advocated various technologies. Additionally, we excluded studies of other health information technologies, such as computerized physician order entry or EHRs that did not explicitly involve the process of HIE. This included studies concerned with intra-organizational information systems. If more than one article reported on findings from the same study, we only included one report in our final sample. We identified only 16 articles that examined HIE use and met our other search criteria. From each article, we extracted how the authors defined usage, the level of analysis for each reported measure (individual, group, organization, or network), and whether usage was reported as a dependent/independent variable or reported descriptively. In addition, from the descriptions provided in the text, we classified the respective HIE architectural strategy using Wilcox and colleagues’ taxonomy.34 This taxonomy identifies six different types of HIE architectural strategies along a functional continuum, ranging from completely separate systems to the use of a common EHR by all organizations participating in the exchange process. Two researchers independently read and coded the 16 articles. Each researcher recorded the level of analysis and HIE architectural strategy for each article. Because a single study might report findings at multiple levels of analysis, we coded each level of analysis as a yes/no to indicate whether that level of analysis was reported or not in the study. Thus, each article had five items to code. Owing to the small sample size, we coded all 16 articles in a single set and then met to compare agreement. After initial coding, we agreed on 56 of 80 (70%) possible coding items. The initial low level of agreement between coders can be attributed to the small sample size, which prevented an opportunity for pilot testing the coding scheme and the lack of details provided by authors regarding study level of analysis. All coding differences were resolved through discussion between coders and reference to source articles.
BODY OF THE REVIEW As evident from table 1, previous researchers have focused mostly on the individual user (seven of 16 studies) and network levels of analysis (10 of 16 studies). Only three studies examined HIE use at the organizational level of analysis. None of the studies specifically reported on group or team level usage measures. While our sample includes studies for several levels of i
The GAO identified information sharing as a benefit of electronic health records in 1991. 303
Review paper Table 1
Articles reporting health information exchange usage by level of analysis and architecture Level of analysis
Article and reference
Individual
Group
Organization
35
Adler-Milstein Branger36 Branger37 Bruun-Rasmussen38 Classen39 Grossman40 Hasman41 Jha42 Johnson43 Kern44 Labkoff19 Liang45 Mattocks46 McDonald47 Overhage17 Wilcox34
Network
HIE architecture
X
HIE technical details not reported Separated systems Separated systems Separated systems Context aware federated Separated federated (portals) Separated systems Multiple (HIE technical details not reported) Centralized data Separated federated (portals) Multiple* HIE technical details not reported Separated federated (portals) Separated federated Separated federated Multiple*
X X X X
X X
X X
X
X X X X
X X
X
X X X
*Labkoff discusses three systems with centralized data and one system in the separated federated model. Wilcox discusses an example system in each architectural category except the monolithic system.
analysis, definitional and conceptual differences between the levels were generally lacking. This was true both for studies that examined usage at only one level of analysis as well as those that described usage at multiple levels within the same study. Table 1 also displays the architectural strategy employed by HIE user organizations. The sample includes studies of four of the six HIE architectural strategies. None of the studies in our sample explicitly reported on systems that fit the separated federated with notification or monolithic system architectural strategies. The systems in our sample included a diversity of exchange methods, such as messaging systems, centralized patient databases, and granting individuals access to other organizations’ information systems. Examinations tended toward HIE efforts with lower levels of functional integration between exchanging organizations. Half of the articles reported on systems with the two lowest levels of integration. Three studies did not include information about the technical details of the HIE systems employed.35 42 45 Two studies included discussion of multiple HIE systems.19 34 Labkoff and Yasnoff studied four different communities engaged in health information infrastructure (HII) initiatives. Three of these communities have created centralized databases for their HIE efforts, while the remaining community is relying on separated federated systems for the participants in their HIE program.19 Wilcox and colleagues provide an example of each architectural strategy except the monolithic system.34 No articles in our sample reported usage measures specific to the monolithic strategy. Table 2 summarizes reported usage measures by level of analysis. The literature includes a diverse set of usage definitions and many studies employed more than one measure of usage. Measures at the individual level, encompassed both the employment and information consumption aspects of HIE usage. In the former category, measures include task related phenomena like access, frequency, time spent, or types of information sought. Whereas these types of measures were most common, they are unfortunately uninformative about individual motivations, or how information was integrated into work processes.25 In contrast, however, two studies applied more informative measures about the application of information into the process of care or on individual cognitions.36 40 Branger and Duisterhout examined workflow impact in the form of the 304
changing distribution of tasks among users and how information changed users’ knowledge about their patients.36 Additionally, a very different measure of deep involvement with an information system is user feedback in the form of modification and new feature requests, which was reported by Grossman and colleagues.40 Finally, more than one study included measures of user satisfaction and usability.36 39 43 While satisfaction has historically been equated with the idea of effectiveness in information systems research, it is really neither effectiveness nor use, but an indicator of an attitude toward the system.48 Likewise usability is not usage, but is an indicator of quality and as such should not be completely dismissed.49 Of the few articles considering usage at the organizational level, the study by Liang and colleagues deserves particular comment.45 It was the only article included in this review to employ a known framework for usage measurement. Massetti and Zmud propose four facets of electronic data interchange usage: breadth (number of exchange partners), volume (amount of information), diversity (types of information), and depth (degree of consolidation between participating organizations).30 Liang and colleagues employ these electronic data interchange (EDI) usage measures in their study of HIE in Beijing hospitals. Even though other studies approached similar ideas about usage, Liang and colleagues benefit from very clearly defined measures and an associated literature base. At the network level of analysis, studies reported measures that represented system employment. At this level, the focus of usage measures changed from the perspective of individual user activity to that of the entire population included in the HIE process. Therefore, measures of access or frequency of use are expressed as the percent of patients or encounters for which the HIE system was employed. Slightly more informative measures, like types and volume of information, occurred frequently at this level and represent the progress an HIE effort has made toward comprehensive information exchange.30 Overall, for the studies in our sample that examined the network level of analysis, none of the conceptualizations of usage provided much insight to how the information is utilized in the process of care or in terms of strategic decision-making. Three studies examined usage as an explanatory variable.17 37 44 Branger and colleagues found that HIE increased J Am Med Inform Assoc 2010;17:302e307. doi:10.1136/jamia.2009.000471
Review paper Table 2 Measures of health information exchange usage in prior research Level
Measure
Individual Access39 44 Additional feature and modification request by individuals40 Diversity of information36 Frequency of access compared to a target level44 Frequency of feature use37 Improved knowledge about patients36 Percent of users accessing system34 40 43 Time processing messages36 Usability43 User satisfaction36 39 Workflow changes36 Group (none) Organizational Breadth (number of exchange partners)45 Depth (technical sophistication)45 Diversity (categories of clinical information provided)45 46 Percent of encounters accessed43 Volume (percent of information exchanged electronically)45 Network Access (per patient basis)34 Diversity (categories of clinical information provided)19 Number of unique patients accessed39 Percent of emergency encounters accessed17 43 Percent of information exchanged38 42 Percent of organizations providing data42 Organizations providing or viewing data35 Volume of information41 47
35 38 47
the communication (number of messages sent and received) between general practitioners and the outpatient clinic of a hospital and the availability of data regarding diagnostics provided at the clinic resulting in improved care for their patients.37 Kern and colleagues found that primary care physicians who used an information portal to review laboratory results at least once over a 6-month period had higher quality of care.44 Overhage and colleagues conduct a controlled trial of HIE among emergency physicians in three hospitals. They found that under some conditions, HIE reduced emergency care charges by $26 at one hospital.17 Most studies in the sample provide reports of descriptive statistics or narratives of HIE use. To a degree, this occurrence is understandable. For many articles in the sample, understanding usage was not the principal line of inquiry. For example, Wilcox and colleagues provide narrative descriptions of various HIE systems as illustrations of each HIE architectural strategy.34 Finally, the collective results from the sample suggest that the existence of an HIE process and associated technology did not guarantee usage. The information systems populated with data from HIE processes were accessed for anywhere from 0.5%17 to 2.6%43 of patient encounters. The percent of patients viewed ranged from 1.1%39 to 10%e20%.34 At the individual level of analysis, only 15% of physicians accessed the system.40
DISCUSSION What do we know about HIE usage? While the diversity of HIE efforts over the past two decades have resulted in various J Am Med Inform Assoc 2010;17:302e307. doi:10.1136/jamia.2009.000471
measures of usage, the literature is neither comprehensive nor systematic. Unfortunately, the bulk of previously utilized measures reported in our sample reveal little information, as many measures were theoretically weak and did not reflect how such usage affected work processes and provider information levels. Usage measures were often not explicitly defined and mostly presented as descriptive statistics. In addition, the architectural strategy for the HIE process was essentially absent from all considerations of usage and while investigations covered multiple architectures and levels of analysis, most researchers ignored the relationship between the two. Nevertheless, the information presented in these studies is of practical importance. Basic measures, like access, at the individual and organizational levels are necessary for understanding adoption, degree of implementation, and even resistance. Also, like many organizations, those engaged in the process of HIE use data for comparisons.50 These observations carry implications for usage definition and measurement. Based on our review of previous HIE research and information system usage research, we suggest future researchers need to move beyond measures of system access to examine more sophisticated measures of HIE use and their effect on the healthcare process. Further, we recommend that researchers, healthcare administrators, and policymakers consider both the level of analysis for HIE use and the architectural strategy in future examinations of HIE efforts.
Understanding meaningful HIE use HITECH requires healthcare providers who use electronic medical records to demonstrate meaningful use of HIE to continue receiving Medicare and Medicaid incentive payments.6 51 We suggest that individuals and collectives interested in understanding HIE use need to apply measures that enable them to capture both aspects of HIE use discussed previously: employment of HIE system features and the application of the information available from the HIE system and associated processes.6 51 The distinction between these two aspects of use go well beyond simply deciding what the numerators and denominators will be included in annual reports submitted to the Centers for Medicare & Medicaid Services. Ultimately, healthcare professionals and organizations are interested in improving the health and wellbeing of their patients. The great potential of HIE is the influence or changes on both individual and organizational-level decision-making around the delivery of care. As noted, the HIE literature already contains examples of information consumption-type usage.36 40 Other use concepts, such as extended use, integrative use, emergent use52 53 and technology interaction behaviors54 from the information systems field would seem to have potential in understanding meaningful HIE use as it pertains to changes in individuals delivery of care and information needs. Any attempts to borrow these later usage constructs from the information systems field will need to operationalize these constructs into carefully defined measures. However, as a matter of practicality, meaningful use cannot be solely defined in terms of organizational or individual impact. Before information from system can be consumed, change decision-making, influence the organization, or have any other type of impact, the system must be employed.49 Therefore, measures of employment have practical place in the evaluation of HIE efforts. To ignore measures of if, when, and how systems were used is to fall into the ‘build it and they will come’ trap. The long history of information technology disuse and failures teaches us we must not assume information system usage or 305
Review paper that the employment of such systems cannot be refined and improved. Again, this review indicates we have numerous candidate measures not only to assess whether or not a provider is making use of the capacity to electronically exchange information but also to evaluate how the systems are being used. Again, as demonstrated by Liang and colleagues, healthcare researchers can benefit from research in the information systems field.45 The four dimensions of EDI use appear as a logical choice to employ in measuring HIE efforts. We encourage others to further investigate the effects of breadth, volume, diversity, and depth of HIE on the healthcare process.
Levels of analysis for HIE use We recommend that future HIE researchers explicitly consider the level of analysis of HIE use in their study. Development of a robust understanding of HIE use will only occur as we build a collective knowledge base that includes the network, organization, group, and individual level of analysis. In policymaking situations such as HITECH requiring HIE use, agencies will likely focus on the individual or organization level of analysis for measuring HIE use. In these cases, usage measures such as breadth, volume, diversity, and depth appropriately applied will enable healthcare providers to demonstrate required usage levels to retain funding. However, the ultimate end user of HIE processes, systems, and information are individuals and, thus, understating HIE use from an individual level of analysis greatly contributes to the collective field of knowledge about HIE use. One goal of HIE is to improve the quality of healthcare provided to each patient.5 As such, understanding HIE use across a group of healthcare providers (regardless of whether or not these providers belong to the same network or organization) and how that collective use contributes improving the healthcare rendered to a patient becomes another crucial part of understanding HIE use. To this end, we call for researchers to investigate HIE system features/processes and how such use influences the healthcare process from multiple levels of analysis.
HIE architectural strategy Through variance in features and purposes, different architectural strategies should constrain or limit the conceptualizations of what constitutes usage and how to measure it. What constitutes usage should not be divorced from how organizations decide to exchange information; the meaning and importance of measures concerning what, how, and when will vary according to the architectural strategy. For example, as one moves along the HIE architectural continuum from separated systems to monolithic systems, the efficiency of communications improves (ie, it takes less time for information about a particular patient to be shared among all providers in a monolithic system than it takes for that same exchange to occur in separated systems among providers). Thus, when one compares the impact of a separated system HIE effort to a monolithic system HIE effort, some observed differences will be inherent to the HIE technology architecture. An additional clear case is where the voluntary aspect of usage is removed and system employment becomes mandatory. Therefore, we recommend that future research not only explicitly defines the HIE technology and processes involved, but specifically relates it to the measure of usage. In our review, we chose to use the taxonomy defined by Wilcox and colleagues.34 We encourage others to follow this strategy in their own research. In some research contexts, this taxonomy could be used as an operationalization of the depth EDI measure. For 306
example, a research examination of HIE efforts in multiple communities or RHIO could use the HIE architectural continuum as a pseudo measure of the sophistication of HIE technical connections between/among healthcare providers in the community or RHIO (cf, Labkoff and Yasnoff19). However, the above architectural constraints only apply to the system employment aspect of usage. The consumption of information, or its use in the decision-making process is not architecturally dependent. While that is the case, architecture and other features of the exchange should not be ignored. Although architecturally independent, consumption of information from an information system can only occur after employment of the information system. If the evaluation standards set under the meaningful usage criteria or in research studies only fall into the consumption or application of information type of usage, it will be all the more difficult to actually understand how the presence of HIE relates to the impact of HIE. A limitation of this review is that we did not consider personal health records (PHRs) as a potential HIE architectural strategy although more than one RHIO/HIE effort exists around PHRs. While a very different architectural strategy, similar considerations of usage and levels are just as applicable to PHRs. The complexity PHRs bring to considerations of usage is the role of the patient as an additional user. PHR measures of usage could potentially expand to include measures of joint usage by patients and providers, similar to the idea of group usage. Most importantly, however, the specific attention to the measurement of usage that we are advocating may help attempts to compare the PHR approach to the other HIE architectures.
CONCLUSION The objective of our recommendations is not review all the possible measures of usage, or the contingent meanings and appropriateness of each based on architectural strategy and level of analysis. We offer these suggestions to highlight the considerable future research agenda in the area of HIE, and to prompt reflection on some practical considerations in the current formulation of national policies. First, HIE usage is not a simple construct, but the choice of its measurement must attend to the users, context, and objectives of the system being examined. Second, HIE usage is not immediately generalizable, but heavily context dependent. Third, conceptualizations of usage available from other disciplines provide a rich potential for the further development and understanding of HIE. For the most part the collective knowledge regarding the impact of HIE efforts toward the improvement of health is lacking. Toward the end of improving patient care, the technologies and processes are necessary, but not sufficient; likewise, usage defined minimally is also a necessary, but insufficient condition. We encourage others to consider our recommendations and incorporate them into improving our understanding of HIE. The provisions of the HITECH act made inquiry into HIE usage more relevant to national healthcare policy; the unpleasant truth that health information technologies are not always used as intended also heightens the importance of understanding usage. HIE usage has been measured variously, but the simple measures of use behavior unfortunately provide little or no insight into the effects of previously inaccessible information on medical decision-making and the process of care. Our understanding of HIE will be advanced greatly by clear conceptions of usage and by utilizing measures that focus on (A) how systems are utilized in practice and (B) what is the application to patient health and wellbeing. J Am Med Inform Assoc 2010;17:302e307. doi:10.1136/jamia.2009.000471
Review paper Competing interests None.
28.
Provenance and peer review Not commissioned; externally peer reviewed. 29.
REFERENCES 1. 2. 3. 4.
5. 6. 7.
8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27.
The National Alliance for Health Information Technology. Report to the office of the national coordinator for health information technology on defining key health information technology terms: department of health & human services, 2008. Acker B, Birnbaum CL, Branden JH, et al. HIM principles in health information exchange. J AHIMA 2007;78:69e74. Brailer DJ. Interoperability: the key to the future health care system. Health Aff 2005;42:w19e21. eHealth Initiative. Emerging trends and issues in health information exchange: select findings from ehealth initative foundation’s second annual survey of state, regional and community-based health information exchange initatives and organizations. Washington, DC: Foundation for eHealth Initative, 2005. Hripcsak G, Kaushal R, Johnson KB, et al. The United hospital fund meeting on evaluating health information exchange. J Biomed Inform 2007;40:S3e10. 111th Congress of the United States of America. American Recovery and Reinvestment Act of 2009. Bush GW. Executive order: incentives for the use of health information technology and establishing the position of the national health information technology coordinator. 2004. http://www.whitehouse.gov/news/releases/2004/04/200404274.html (cited 2008 Oct 20). Devaraj S, Kohli R. Performance impacts of information technology: is actual usage the missing link. Manage Sci 2003;49:273e89. Jasperson J, Carter PE, Zmud RW. A comprehensive conceptualization of postadoptive behaviors associated with information technology enabled work systems. MIS Quarterly 2005;29:525e57. Adler-Milstein J, Bates DW, Jha AK. U.S. regional health information organizations: progress and challenges. Health Aff 2009;28:483e92. Chin T. Doctors pull plug on paperless system. Am Med News 2003. http://www. ama-assn.org/amednews/2003/02/17/bil20217.htm (accessed Feb 2010). Dagroso D, Williams PD, Chesney JD, et al. Implementation of an obstetrics EMR module: overcoming user dissatisfaction. J Healthc Inf Manag 2007;21:87e94. Goldfinch S. Pessimism, computer failure, and information systems development in the public sector. Public Administration Review 2007;67:917e29. Kaplan B, Harris–Salamone KD. Health IT project success and failure: recommendations from literature and an AMIA workshop. J Am Med Inform Assoc 2009;16:291e9. Larum H, Ellingsen G, Faxvaag A. Doctors’ use of electronic medical records systems in hospitals: cross sectional survey. BMJ 2001;323:1344e8. Lorenzi NM. Strategies for creating successful local health information infrastructure inatiatives, Nashville, TN: Vanderbilt University, 2003. Overhage J, Deter P, Perkins S, et al. A randomized, controlled trial of clinical information shared from another institution. Ann Emerg Med 2002;39:14e23. Rubin RD. The community health information movement: where it’s been, where it’s going. In: O’Carroll PW, Yasnoff WA, Ward ME, et al, eds. Public health informatics & information systems. New York: Springer, 2003. Labkoff SE, Yasnoff WA. A framework for systematic evaluation of health information infrastructure progress in communities. J Biomed Inform 2007;40:100e5. Cusack CM, Poon EG. Health information exchange toolkit. Rockville, MD: Agency for Healthcare Research & Quality, 2007. Healthcare Information & Management Systems Society. RHIO/HIE: definitions & acronyms. 2008. http://www.himss.org/content/files/RHIO_Definitions_Acronyms. pdf (cited 2008 July 3) Ives B, Hamilton S, Gordon BD. A framework for research in computer-based management information systems. Manage Sci 1980;26:910e34. Hakkinen H, Turunen P, Spil T. Information in health care process - evaluation toolkit development. system sciences, 2003 proceedings of the 36th Annual Hawaii International Conference on; 2003; 2003. 7. Doll WJ, Torkzadeh G. Developing a multidimensional measure of system-use in an organizational context. Inf Manage 1998;33:171e85. Burton-Jones A, Straub DW. Reconceptualizing system usage: an approach and empirical test. Information Systems Research 2006;17:228e46. DeLone WH, McLean ER. Information systems success: the quest for the dependent variable. Information Systems Research 1992;3:60e95. eHealth Initiative. Improving The Quality Of Healthcare Through Health Information Exchange: Selected Findings from eHealth Initiative’s Third Annual Survey of Health Information Exchange Activities at the State. Washington, DC: Regional and Local Levels, 2006.
J Am Med Inform Assoc 2010;17:302e307. doi:10.1136/jamia.2009.000471
30. 31. 32.
33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53.
54.
Burton-Jones A, Gallivan MJ. Toward a deeper understanding of system usage in organizations: a multilevel perspective. MIS Quarterly 2007;31:657e79. Straub D, Limayem M, Karahanna-Evaristo E. Measuring System Usage: Implications for IS Theory Testing. Manage Sci 1995;41:1328e42. Massetti B, Zmud RW. Measuring the extent of EDI usage in complex organizations: strategies and illustrative examples. MIS Quarterly 1996;20:331e45. Bodenheimer T. Coordinating care e a perilous journey through the health care System. N Engl J Med 2008;358:1064e71. Gerteis M. Coordinating care and integrating services. In: Gerteis M, Edgman-Levitan S, Daley J, Delbanco TL, eds. Through the patient’s eyes: understanding and promoting patient-centered care. San Francisco, CA: Jossey-Bass, 1993. Schoen C, Osborn R, Huynh PT, et al. Primary care and health system performance: adults’ experiences in five countries. Health Aff 2004;23: w487e503. Wilcox A, Kuperman G, Dorr DA, et al. Architectural strategies and issues with health information exchange. AMIA Annu Symp Proc 2006;2006:814e18. Adler-Milstein J, McAfee AP, Bates DW, et al. The State Of Regional Health information organizations: current activities and financing. Health Aff 2008;27: w60e9. Branger PJ, Duisterhout JS. Electronic data interchange in medical care: an evaluation study. Proc Annu Symp Comput Appl Med Care 1991:58e62. Branger PJ, van’t Hooft A, van der Wouden JC, et al. Shared care for diabetes: supporting communication between primary and secondary care. Int J Med Inform 1999;53:133e42. Bruun-Rasmussen M, Bernstein K, Vari S, et al. Regional health care networks. Stud Health Technol Inform 1999;68:265e9. Classen DC, Kanhouwa M, Will D, et al. The patient safety institute demonstration project: a model for implementing a local health information infrastructure. J Healthc Inf Manag 2005;19:75e86. Grossman JM, Bodenheimer TS, McKenzie K. MARKETWATCH: Hospital-Physician Portals: The Role Of Competition In Driving Clinical Data Exchange. Health Affairs 2006;25:1629e36. Hasman A, Ament A, Arnou PC, et al. Inter–institutional information exchange in healthcare. Int J Biomed Comput 1992;31:5e16. Jha AK, Doolan D, Grandt D, et al. The use of health information technology in seven nations. Int J Med Inform 2008;77:848e54. Johnson KB, Gadd C, Aronsky D, et al. The MidSouth ehealth alliance: use and impact in the first year. AMIA Annu Symp Proc 2008;2008:333e7. Kern LM, Barron Y, Blair AJ, et al. Electronic result viewing and quality of care in small group practices. J Gen Intern Med 2008;23:405e10. Liang HG, Xue YJ, Byrd TA, et al. Electronic data interchange usage in China’s healthcare organizations: the case of Beijing’s hospitals. Int J Inf Manage 2004;24:507e22. Mattocks K, Lalime K, Tate JP, et al. The state of physician office-based health information technology in Connecticut: current use, barriers and future plans. Conn Med 2007;71:27e31. McDonald CJ, Overhage JM, Barnes M, et al. The Indiana network for patient care: a working local health information infrastructure. Health Aff 2005;24:1214e20. Melone NP. A theoretical assessment of the user-satisfaction construct in information systems research. Manage Sci 1990;36:76e91. DeLone WH, McLean ER. The DeLone and McLean Model of Information Systems Success: a ten-year update. J Management Information Systems M.E. Sharpe Inc. 2003:9e30. eHealth Initative. eHI Foundation launches major survey of state, regional and community health information exchange initiatives. 2005. http://www. ehealthinitiative.org/news/2005-05-23.mspx (cited 2008 July 12). Seidman J. The big gamble: will stimulus dollars pay off in health information consumers can use? J AHIMA 2009;80:34e6. Cooper RB, Zmud RW. Information technology implementation research: a technological diffusion approach. Manage Sci 1990;36:123e39. Saga VL, Zmud RW. The nature and determinants of IT acceptance, routinization, and infusion. In: Levine L, ed. Diffusion, transfer and implementation of information technology. Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 1994:67e86. Barki H, Titah R, Boffo C. Information system use-related activity: an expanded Behavioral conceptualization of individual-level information system use. Information Systems Research 2007;18:173e92.
307
