Evaluating PACS Success: A Multidimensional Model Guy Paré, Ph.D. David Aubry, M.S. HEC Montréal
[email protected] [email protected] Luigi Lepanto, M.D. Centre Hospitalier de l’Université de Montréal
[email protected] Claude Sicotte, Ph.D. University of Montréal
[email protected] Abstract A picture archiving and communications system (PACS) is an integrated workflow system for managing images and related data which is designed to streamline operations throughout the whole patient care delivery process. PACS has become a mature technology over the past few years and has been widely implemented in several developed countries. Evaluation of PACS success is a major challenge to healthcare organizations. A review of previous PACS research suggests a fragmented and focused evaluation approach, thus offering limited discussion of comprehensive views of PACS success or systematic and practical guidance to its evaluations. Based on two prevalent information systems success models, this paper proposes and describes an integrated framework for evaluating PACS success in hospital settings. It details the validation process of the proposed model and its related measurement instrument at a large tertiary-care teaching hospital in Canada. Future research directions that extend the proposed research model are highlighted.
1. Introduction PACS has become an important component of many radiology departments and hospitals around the world [1]. A large number of studies have attempted to identify those factors that contribute to PACS success [2, 3]. Results from these studies [4, 5] clearly reveal that the ultimate success of PACS requires healthcare organizations and managers to adequately address various types of challenges: technological (e.g., integration with other information systems), managerial (e.g., project management), organizational (e.g., availability of
resources), behavioural (e.g., change management) and political (e.g., alignment among key participants). However, the dependent variable in these studies – PACS success – has been an elusive one to define. Different researchers have addressed different aspects of success, making comparisons difficult and the prospect of building a cumulative tradition for research similarly elusive. Most investigations have considered a single or at best a small number of factors, contributing to a fragmented view of PACS success. Broadly, these studies may be classified into those that consider the impact of PACS on radiologists’ workload and productivity [6], those that consider its clinical implications [7] and those associated with performance of the radiology department [8]. In short, past empirical and evaluative studies have provided limited discussion of conceptual frameworks for holistic or comprehensive understanding of PACS success or systematic and practical guidance to its operationalization. To organize this diverse research, as well as to present a more integrated view of the construct of PACS success, a comprehensive success model is introduced. Our aim is to synthesize previous research into a more coherent body of knowledge and to provide guidance to managers, clinicians, and researchers. Evaluation of system success or effectiveness has been a fundamental issue and dominant focus in information systems (IS) research over the past 30 years. Since PACS is a particular or specialized form of system, a logical and reasonable departure point for evaluating PACS success is the relevant IS literature. From this perspective, the proposed multidimensional model of PACS success is based on DeLone and McLean’s IS success framework [9, 10] which has emerged to be a dominant model for system evaluation research. Our integrated model also comprises key constructs from a complementary
framework, namely, Battacherjee’s IS continuance model [11]. The remainder of this paper is organized as follows. Section 2 reviews relevant previous PACS evaluation studies and highlights the research motivation. Section 3 provides an overview of Delone and McLean’s success framework as well as that of Battacherjee and concludes with the presentation of the resulting model for evaluating PACS success. Section 4 describes the operationalization of the success model in terms of methods and measures. In section 5, we detail the adopted research design and methods in order to validate the proposed model and the related measures, as well as test the strength of the relationships between variables. Data collection, which takes place at a large tertiary-care teaching hospital in Canada, is currently in its final phase. The paper concludes with a summary and discussion of future research in section 6.
2. Literature review and research motivation Evaluation represents a critical issue in PACS research and practice. Unfortunately, in searching for a PACS success measure, rather than finding none, there are nearly as many measures as there are studies. The reason for this is understandable when one considers that “information,” as the output of an information system such as a PACS, can be measured at different levels, including the technical level, the semantic level and the effectiveness level. Shannon and Weaver [12] defined the technical level as the accuracy and efficiency of the system which produces the information, the semantic level as the success of the information in conveying the intended meaning, and the effectiveness level as the effect of the information on the receiver. As reported in DeLone and McLean [9], the three levels of information of Shannon and Weaver are shown to yield six distinct categories or aspects of PACS success. As shown in Table 1, these categories are system quality, information quality, use, user satisfaction, individual impact, and organizational impact. Looking at the first of these categories, some researchers have chosen to focus on the desired attributes of the PACS itself. For instance, Cox and Dawe [13] examined the speed of image availability, the ease of use of the system, and the frequency of system breakdown. Gale et al. [6] evaluated several aspects of the quality of a PACS interface design. For their part, Tucker et al. [14] considered several image issues, integration of PACS with radiology information system (RIS) and hospital (HIS) information systems to be an important component of PACS success or failure. Rather than measure the quality of the PACS performance, other researchers have preferred to focus on the quality of the information that the system produces, primarily in the form of images and reports. For instance,
Lou et al. [15] considered the data integrity and completeness of acquired images. High quality images in terms of timeliness, accuracy, completeness, etc. were also considered to be a key success factor in several evaluative studies [13, 16, 17, 18]. Table 1. Categories of PACS success Shannon & Weaver Categories of IS success Technical level Semantic level Influence level
System quality Information quality Use | User satisfaction | Individual impact | Organizational impact
Another dimension of quality, not presented in Table 1, has also been studied in prior PACS research, namely, service or support quality. Indeed, a few investigators found that the technical support is a key aspect of PACS success; regardless of whether this service is delivered by an internal IS department or outsourced to an external service provider. Technically competent staff and a realistic time commitment were found to be associated with PACS success [4, 13, 14]. At the influence level, some researchers have analysed the interaction of the information project with its recipients, namely, the radiologists, technologists and clinicians, by measuring use and user satisfaction. Fundamentally, the use of a PACS is central to its success. Some studies have computed actual use (as opposed to reported use) by radiologists, technologists and clinicians through hardware monitoring which have recorded the connected time per day and the number of study retrievals [19], the number of system functions utilized and the amount of time spent interpreting/reviewing images [20]. Perceived usage was also considered in several evaluative studies. For instance, Tamm et al. [21] asked oncology clinicians the number of studies and radiology reports they viewed per week and the amount of time they spent reviewing each study’s image. With regard to user satisfaction, Bryan et al. [22] studied the major causes of radiologists’ satisfaction and dissatisfaction (frustration) with the PACS. Users’ expectations have also been studied widely. For example, Baumann and Gell [4] conducted a longitudinal survey of 1,000 facilities around the world. Clinicians reported that their expectations of the PACS had been met in 81% of cases and 97% of the users would recommend PACS to others. More recently, Pilling [17] assessed the acceptability to radiologists of a PACS. Respondents judged that PACS had made a
positive change in their working practices and had met their expectations. Undeniably, impacts, whether at the individual or the organizational level, represent the most widely used construct of PACS success. A vast majority of researchers have been interested in the influence which the PACS has on the users. For instance, Bryan et al. [22], Kato et al. [23] and Reiner et al. [3] investigated the impact of PACS on radiologists’ productivity and report/ interpretation time. Hertzberg et al. [7] examined the relative accuracy of interpretation of sonography when viewed on a PACS workstation or on film. Other researchers have studied the influence of PACS on technologists’ productivity. For instance, Reiner and Siegel [24] assessed the impact of filmless operation and computed radiography on technologist’s examination times compared with conventional film-screen radiography. Finally, researchers have been concerned with workflow and other performance issues. For instance, Hayt et al. [8] studied the impact of a PACS on radiology operations and service at a large urban hospital. Precisely, they found that, with the aid of a PACS, the hospital gained complete control of a runaway film problem and report turnaround time changed from being completely unacceptable to acceptable. In the same vein, Mattern et al. [25], Pavlicek et al. [16] and Weatherburn et al. [26] examined the impact of electronic imaging on several outcome measures including image reject rates, time to final diagnosis, time to final treatment and need for follow-up. Blado et al. [27], collected data on rejected images and images from repeated examinations. Dackiewicz et al. [28] examined the influence of digital radiography on clinical workflow and patient satisfaction. Redfern et al. [29] evaluated the relationship between patient volume and workflow for radiologists who began to interpret images from multiple clinical sites after the introduction of a PACS. Reiner et al. [30] studied the impact of filmless operation on the relative frequency of in-person consultations in the radiology department between radiologists and clinicians. As a final example, Weatherburn and Bryan [31] examined whether the doses for the radiographic examination of the lateral lumbar spine changed as a result of the introduction of a hospitalwide PACS. In conclusion, once this expanded, but rather fragmented view of PACS success is recognized, it is not surprising to find that there are so many different measures of this success in the literature depending upon which aspect of PACS the researcher focused his or her attention. As mentioned earlier, a review of relevant prior IS success frameworks may shed light on the needed comprehensive view of PACS success as well as systematic and practical guidance for its evaluations. In particular, the IS success framework by DeLone and McLean [10] as well as Chattaberjee’s IS continuance model [11] have emerged as important and prevalent
frameworks for evaluating IS success. The following section describes both models upon which the proposed integrative model of PACS success is developed.
3. An integrated model of PACS success Motivated by the need for a comprehensive framework for advancing and integrating IS research findings, DeLone and McLean [9] postulated an IS success framework. Based on the communication theory of Shannon and Weaver [12], the information influence theory of Mason [32], and a fairly comprehensive synthesis of the important system evaluation research conducted between 1981 and 1987, the original model, which was published in 1992, offers a multidimensional leans to IS success and, at the same time, singles out a set of common measurements for each success dimension. Since the publication of the original framework, about 300 articles in refereed journals have referred to, and made use of, this IS success model. Several empirical studies explicitly tested the relationships among the variables identified in the original model [33, 34, 35]. Yet, other studies have implicitly tested the model by investigating multiple success dimensions and their interrelationships [36, 37]. Taken as a whole, these studies gave strong support for the proposed associations among the IS dimensions and helped to confirm the causal structure in the model. Judged by its frequent citations in articles published in leading IS journals, this framework has become a dominant evaluation model in IS research. Based on research contributions since the publication of the model, DeLone and McLean have updated their original success framework in 2003 (see Figure 1). SYSTEM QUALITY
USAGE
INFORMATION
NET BENEFITS
QUALITY USER SATISFACTION SERVICE QUALITY
Figure 1. DeLone and McLean’s revised success model The model in Figure 1 indicates that success of an information system is multi-dimensional and can be represented by the quality characteristics of the system itself (SYSTEM QUALITY); the quality of the output (INFORMATION QUALITY); the quality of the technical support or service (SERVICE QUALITY); the
consumption of the output of the system (USAGE); the user’s response to the system (USER SATISFACTION); and, ultimately, the impacts the system has (NET BENEFITS). DeLone and McLean’s revised model makes two important contributions to the understanding of IS success. First, it provides a scheme or a framework for categorizing the multitude of IS success measures that have been used in the literature. Second, it suggests a model of temporal and causal interdependencies between the categories. Two constructs were added to DeLone and McLean’s model to recognize complementary research findings in the IS field. DeLone and McLean are primarily concerned with acceptance behaviours, namely, use (or intention to use). While usage represents an important indicator of system success, long-term viability and its eventual success depend on its continued use. As explained by Bhattacherjee [11], IS continuance is not an alien concept in IS research. Indeed, many studies have acknowledged the existence of a post-acceptance stage when IS use transcends conscious behaviour and becomes part of normal routine activity. Innovation diffusion theory suggests that adopters eventually reevaluate their earlier acceptance decision and decide whether to continue or discontinue using an innovation [38]. In line with such reasoning, like Bhattacherjee, we think it is important to differentiate between acceptance and continuance behaviours and, hence, we include system continuance intention as the ultimate dependent variable in our own success model (see Figure 2). The model tested by Battacherjee [11] is based on expectation-confirmation theory (ECT) [39]. Per ECT, users’ IS continuance intention is determined primarily by their satisfaction with prior IS use and their perceived usefulness of IS use (perceived net benefits). Therefore, as shown in Figure 2, both user satisfaction and net benefits are associated with system continuance. Lastly, we posit that PACS continuance intention is influenced (both directly and indirectly) by another construct, namely, confirmation of expectations following actual use of the system. Confirmation is positively related with system continuance (and user satisfaction) because it implies the realization of the expected benefits of IS use, while disconfirmation denotes failure to achieve expectation [11]. As depicted in Figure 2, the resulting model comprises eight interrelated dimensions of PACS success: perceived system quality, perceived information/image quality, perceived service quality, system use, user satisfaction, perceived net benefits, confirmed expectations, and system continuance intention. “System quality,” in a PACS environment, measures the desired characteristics of a PACS such as its reliability, ease of use, availability, security, and response time. Information/image quality captures the content issue
of a PACS. Patient information and images produced by a PACS must be precise, understandable, complete, and available on time, to name a few, if we expect radiologists and other groups of adopters to use it. “Service quality,” the overall support delivered by the service provider, applies regardless of whether this support is delivered by the internal IT department or outsourced to a PACS service provider. Consistent with that commonly defined clinically, service quality can be examined in terms of service consistency, reliability, timeliness, empathy, assurance, and accuracy or adequacy. PERCEIVED SYSTEM QUALITY
H14 (NOTTESTED) H1 H2
PERCEIVED INFORMATION QUALITY
SYSTEM USAGE
H3
PERCEIVED NETBENEFITS
H7 H4 H5
PERCEIVED SERVICE QUALITY
H9
USER SATISFACTION
H12
SYSTEM CONTINUANCE INTENTION
H10 H11
H6
H13 H8 CONFIRMED EXPECTATIONS
Figure 2. An integrated model of PACS success. Next, we concur with DeLone and McLean that “system usage” is an appropriate indicator of success in most IT implementation projects, and PACS is no exception. “User satisfaction” remains an important means of measuring users’ opinions about PACS. “Net benefits” are the most important success measures as they capture the balance of positive and negative impacts of PACS on radiologists, technologists, physicians and hospitals in general. “Net benefits” success measures are most important, but they cannot be analyzed and understood without “system quality,” “information/image quality,” and “service quality.” Next, as explained above, “confirmed expectations” are positively related to “user satisfaction” with PACS use because it implies realization of the expected benefits of the system. Lastly, while initial acceptance (usage) of PACS is an important first step toward realizing success, long-term viability of a PACS and its eventual success depends on its continued use “system continuance” rather than first-time use. It is then hypothesized that system continuance will be positively associated with user satisfaction, net benefits and confirmed expectations. In short, the hypotheses that follow directly from the proposed model are summarized in Table 2:
H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14
Table 2. Research hypotheses. Use of PACS is positively associated with perceived quality of the system. Users are more satisfied with PACS of higher perceived (system) quality. Use of PACS is positively associated with perceived information quality. Users are more satisfied with PACS of higher information quality. Use of PACS is positively associated with perceived service quality. User satisfaction is positively associated with PACS service quality. Levels of user satisfaction and levels of PACS use are mutually and positively associated. Users’ extent of confirmation is positively associated with their satisfaction with PACS use. Perceived net benefits are positively associated with PACS use. Perceived net benefits are positively associated with user satisfaction. Users’ level of satisfaction with PACS usage is positively associated with their PACS continuance intention. Perceived net benefits are positively associated with users’ PACS continuance intention. Users’ extent of confirmation is positively associated with their PACS continuance intention. Users’ PACS continuance intention is positively related to their future use of the system (not tested in this study).
4. Operationalization of the proposed model The following step consisted in reviewing the IT and digital imaging literature in search of specific perceptual measures for the various constructs included in the conceptual model. In and of itself, perceived system quality is a multidimensional construct. For one thing, it comprises ease of use, that is, the extent to which learning and using a system is free from effort. Perceived system sophistication represents another key aspect of system quality. Put simply, it represents the perceived diversity and quality of functionalities offered by the PACS. Ease of access to PACS, both onsite and offsite, represents another dimension of perceived system quality. Next, reliability of the hardware and software components of a PACS is also key to its success. Reliability mainly refers to perceived frequency of system failures and breakdowns (hardware component) as well as perceived number of “bugs” contained in the system (software component). Response time (in terms of image downloading and
visualizing) represents another dimension of system quality. Extent of PACS integration with RIS and other hospital information systems has also been identified as a key dimension of PACS quality [14]. Lastly, security also represents a fundamental aspect of PACS quality when one considers that the overall damages and costs associated with a destroyed PACS archive storage and server is comparable to losing the entire onsite film archive of the hospital department. As a consequence, adequate security procedures should include both data redundancy as well as PACS data recovery [40]. In short, all measures developed for the system quality construct were original scales, except for the ease of use dimension which was adapted from Seddon and Kiew [34]. Perceived information quality refers to the quality of the images and patient information produced by the PACS. Several aspects are essential with regard to information quality, namely, timeliness, accuracy, completeness, ease of understanding or interpretability, and relevance [27, 41]. The scale used to measure the quality of the images and information generated by a PACS was adapted from Doll and Torkzadeh [42]. The overall quality of the images produced by the system was also assessed using Pilling’s original measure [17]. Service quality, in the context of PACS implementation, refers to the perceived quality of the support and service provided by the provider of the system and/or the internal staff responsible for PACS support and maintenance. This construct was measured using a 13-item scale developed in the marketing area [43] and then adapted to the IT context [44]. PACS usage, from a perceptual standpoint, can be measured using a variety of dimensions and measures common to technology acceptance or adoption studies. First, intensity of use is frequently used as a measure of system success. By reflecting the amount of time engaged with the technology, intensity clearly relates to the technology’s degree of embeddedness. We propose to measure it as the amount of time spent using the system (as a self-reported value). Precisely, we ask respondents to indicate the average number of hours they spend using the PACS per week and which percentage this time represents of their work. Such measure was adapted from Seddon and Kiew [34]. Second, frequency of use is also often used as a success criterion. Frequency of use was adapted from Raymond [45]. This measure provides a perspective of use slightly different than time. It is measured on a seven-point Likert-type scale ranging from "less than once a day" to "several times a day". A third dimension refers to the various PACS functionalities or features used by the users (e.g., access to images from home via Internet; split screen functionality; add features). Scope is therefore defined as the degree to which the PACS is used for a variety of purposes. It is based on the theory that as an individual appropriates a technology for more purposes it becomes a
greater part of that individual’s work system [46]. A new scale was developed in order to capture this dimension of PACS usage. Next, user satisfaction refers to the degree to which a user is satisfied with his or her overall use of the PACS. Collective findings from prior IS research have suggested that user satisfaction is a strong and critical manifestation of systems success. A four-item scale developed by Battacherjee [11] was adapted to the PACS context to measure radiologists’, technologists’ as well as physicians’ satisfaction with the system. This measure has demonstrated high psychometric qualities in prior IS studies. Perceived net benefits represent another multidimensional success construct. To evaluate the users’ perception of the impacts of the PACS, their views were sought on whether there was a comparative improvement, pre- and post-PACS deployment. Table 3 synthesizes the various benefits radiologists, technologists, and clinicians were asked to rate. Table 3. Measures of perceived net benefits. Perceived net benefits R T C Speed of image availability ● ● Number of lost images ● ● Number of unread studies ● ● Speed of clinical decision making ● ● Overall report turnaround time ● ● Number of repeated examinations ● ● ● Number of rejected images ● ● Time devoted to image searching ● ● Time devoted to quality control ● Accuracy of diagnoses ● ● Number of patients who move through ● the procedure room per hour Overall personal productivity ● ● ● Radiologists/clinicians relations ● ● Clinicians/patients relations ● Overall quality of patient care ● ● Quality of work life ● ● ● Legend: R= radiologist survey; T= technologist survey; C= clinician survey. The questions that emerged from the instrument development phase use various formats, including:
Seven-point Likert scales, in asking for judgements on factors such as perceived speed of clinical decision making; Simple quantitative responses to questions like how much time does PACS save you; Open-ended questions, to obtain a broader perspective on the respondents’ views about the benefits of PACS.
Next, confirmed expectations basically refer to the users’ perception of the congruence between expectation of PACS use and its actual performance. This construct was measured using a three-item scale adapted from Bhattacherjee [11]. Lastly, system continuance intention, that is, users’ intention to continue using the PACS, was measured using a two-item scale also developed by Batthacherjee [11].
5. Methodology In order to test the content validity of the proposed research model, a series of in-depth interviews were first conducted with representative respondents at the Centre Hospitalier de l’Université de Montréal (CHUM) where a PACS was implemented throughout the year 2002. The academic medical center has over 1,400 licensed beds housed in three distinct campuses. CHUM counts over 900 physicians, 47 radiologists, and over 150 radiology technologists. Lastly, over 365,000 radiology exams are produced each year. Given that different stakeholders, having different needs and interests, may attribute different outcomes to the PACS, may ignore outcomes they don’t want to think about, and may evaluate the “same” outcomes differently, interviews were then conducted with twelve representatives from various groups of users (radiologists, physicians and technologists) and PACS managers. Conclusively, the overall success model shown in Figure 2 appeared to characterize well the reality of PACS success in the hospital context. Indeed, all variables included in the model were identified by at least one respondent although the most referenced or cited dimensions included system quality, information/image quality, user satisfaction and net benefits. Next, in order to refine further our questionnaire instruments, a pre-test was administered to a relatively small number of potential respondents. The primary objective was to have additional feedback on the content of each measure before distribution to the potential respondents. Interviews were then conducted with three residents. All three reviewers were very thorough in their comments and several suggestions were offered to improve the wording of the scales. In fact, most of the changes made specifically affected the format of the instruments without affecting its substance. A full-scale survey was recently conducted at the CHUM in order to assess the reliability and validity of our success measures as well as the strength of the relationships between the various constructs. As explained earlier, a distinct questionnaire was then built for radiologists (n=47), technologists (n=160), and clinicians (n=649). All measures included in the three versions of the questionnaire were identical except for the usage (scope) and net benefits dimensions whose items were tailored to each group of respondents.
Subjects received through internal mail a packet that contained a cover letter, a questionnaire, and a return envelope. Participation was voluntary, and respondents were assured that their individual responses would be treated as confidential. Four weeks later, a reminder letter was sent to all participants, requesting that those who had not yet participated complete the questionnaire. As of today, a total of 232 questionnaires were returned to the researchers (27% return rate). While this is lower than desired, it is not unusual for large scale surveys. Among the returned questionnaires, 24 (51% response rate) were completed by radiologists, 77 (48% response rate) by technologists, and 131 (20% response rate) by physicians. Note that fourteen questionnaires returned by physicians were removed from our database due to missing data, leaving us with a final sample of 218 responses. With regard to data analysis, composite reliability coefficients (Cronbach alpha) have been computed in order to assess the internal consistency of each scale. Table 4 presents the results associated with the assessment of the internal consistency of each scale. The composite reliability coefficients of all the scales, but one, ranged from 0.72 to 0.97. In the coming weeks, a principal components factor analysis will be performed to check for construct validity. Descriptive statistics will also be computed and ANOVA and t-test will be used to test for significance between groups of PACS users. Hypothesis testing will then be performed using linear regression analyses. Lastly, qualitative analysis of the open-ended questions included in the three survey instruments will also be conducted for triangulation purposes. All quantitative and qualitative results will be presented at the conference.
model improvement and refinement. Further model validation will also be important. One possible direction is to validate and thus enhance the model by conducting other case studies of PACS success in various hospital and clinical settings. Eventually, empirical tests of the model will be essential. Towards, this, survey studies that target various users and user groups in hospitals, clinics, and other clinical settings are desirable.
6. Concluding remarks
7. References
Rapidly, PACS technology is becoming a reality in many North American, European, and Asian hospitals. Amid the growing interest in and implementation of PACS around the world, it is essential to address the challenge in evaluation of PACS success. A review of the digital imaging literature revealed a fragmented evaluation approach, most studies focusing on a single or a small group of success measures. As a consequence, discussion on comprehensive views of system success or systematic guidance to its evaluations is limited. Based on the IS success model by DeLone and McLean as well as the IS continuance model of Battacherjee, this paper proposed a multidimensional, integrated model for evaluating PACS success from multiple stakeholders’ perspectives. The research reported here signifies an important first step toward a comprehensive and holistic understanding of PACS technology success in the hospital setting. Continued research will be needed in several areas. First, additional analysis and synthesis efforts are needed for
[1] H.K. Huang, PACS – Basic Principles and Applications, John Wiley & Sons, New York: NY, 1999.
Table 4. Internal consistency results. # of Construct Alpha items System Quality Ease of use 8 .91 Sophistication 3 .83 Integration 1 Reliability 5 .86 Ease of access 3 .79 Interface quality 6 .78 Rapidity 6 .78 Information / Information quality 6 .91 Image Quality Image quality 8 .91 Service Quality 13 .97 Usage Intensity 2 .83 Frequency 1 Scope - Radiologists 8 .66 Scope - Physicians 6 .72 User Satisfaction 4 .94 Net Benefits Radiologists and 20 .80 physicians1 Technologists 6 .92 Confirmed Expectations 2 .77 System Continuance Intention 3 .82
[2] M.C. Trudel and G. Paré, "Barrières à l’innovation technologique dans les hôpitaux : étude du premier cas d’adoption du PACS au Québec", Gestion- Revue international de gestion, 29(1), 2004, pp.36-45. [3] B.I. Reiner, E.L. Siegel and K. Sidiqqui, "Evolution of the Digital Revolution: A Radiologist Perspective," Journal of Digital Imaging, 16(4), December 2003, pp.324-330.
1
A combined measure was developed in order to satisfy the usual requirement of at least five times as many respondents as items [47]. We intend to validate the success model in Figure 2 in other hospital settings and collect sufficient data as to test the reliability of our original measures (radiologists: 23 items; physicians: 27 items).
[4] R.A. Bauman and G. Gell, "The Reality of PACS: A Survey," Journal of Digital Imaging, 13(4), November 2000, pp.157-169.
[18] M.E. Blado and A.B. Tomlinson, "Monitoring the Accuracy of a PACS Image Database," Journal of Digital Imaging, 15 (supplement 1), 2002, pp.87-95.
[5] M.D. Ralston and R. Coleman, "Sharing of a single arching and communications system among disparate institutions: Barriers to success," Journal of Digital Imaging, 15 (supplement 1), 2000, pp.3-6.
[19] K.W. McEnery, et al., "Evaluation of Soft Copy Distribution of Diagnostic Studies before Soft Copy Interpretation," Journal of Digital Imaging, 15(1), 2002, pp.76-80.
[6] D.R. Gale, M.E. Gale, R.K. Schwartz, V.V. Muse and S.T. Walker, "An Automated PACS Workstation Interface: A Timesaving Enhancement," American Journal of Radiology, 174, January 2000, pp.33-36
[20] H.K. Huang, S.L. Lou and W.P. Dillon, "Neuroradiology Workstation Reading in an Inter-hospital Environment: a Nineteen Month Study," Computerized Medical Imaging and Graphics, 21(5), 1997, pp.309-317.
[7] B.S. Hertzberg, et al. "PACS in Sonography: Accuracy of interpretation using film compared with monitor display," American Journal of Radiology, 173, November 1999, pp.1175-1179.
[21] C.K. Tamm, et al., "Measure of the Utility of a Clinical PACS: Comparison of Self-Reported Measures and Direct Measures of PACS Usage by Clinicians," Journal of Digital Imaging 16(1), 2003, pp.58-60.
[8] D.B. Hayt, "Filmless in 60 days: The Impact of PACS within a large urban hospital," Journal of Digital Imaging, 14(2), June 2001, pp.62-71.
[22] S. Bryan, et al., "The Benefits of Hospital-wide PACS: a Survey of Clinical Users or Radiology Services," The British Journal of Radiology, 72, 1999, pp.469-478.
[9] W.H. DeLone and E.R. McLean, "Information Systems Success: The Quest for the Dependent Variable," Information Systems Research, 3(1) March 1992, pp.60-95.
[23] H. Kato, et al., "Preliminary time-flow study: comparison of interpretation times between PACS workstations and films," Computerized Medical Imaging and Graphics, 19(3), 1995, pp.261-265.
[10] W.H. DeLone and E.R. McLean, "The DeLone and McLean Model of Information Systems Success: A Tenyear Update," Journal of MIS, 19(4), Spring 2003, pp.930. [11] A. Battacherjee, "Understanding information systems continuance: an expectation-confirmation model," MIS Quarterly, 25(3), September 2001, pp.351-370. [12] Shannon, C.E. and Weaver, W., The Mathematical Theory of Communication, University of Illinois Press, Urbana: IL, 1949. [13] B. Cox and N. Dawe, "Evaluation of a PACS System on an Intensive Care Unit," Journal of Management in Medicine, 16(2&3), pp.199-205.
[24] B.I. Reiner and E.L. Siegel, "Technologists’ productivity when using PACS: Comparison of film-based versus filmless radiography," American Journal of Radiology, 179, July 2002, pp.33-37. [25] C.W.T. Mattern, et al. "Electronic Imaging Impact on Image and Report Turnaround Times," Journal of Digital Imaging, 12(2), May 1999, pp.155-159. [26] G.C. Weatherburn, et al., "A comparison of image reject rates when using film, hard copy computed radiography and soft copy images on PACS workstations," The British Journal of Radiology, 72, 1999, pp.653-660.
[14] D.M. Tucker, G.T. Barnes and R.E. Koehler, "PACS in the ICU," Radiology, 196, 1995, pp.297-304.
[27] M.E. Blado, Y. Ma, R.a. Corwin and S.G. Carr, "Impact of Repeat/Reject Analysis in PACS," Journal of Digital Imaging, 16 (Supp. 1), 2002, pp.22-26.
[15] S.L. Lou, "An automated PACS image acquisition and recovery scheme for image integrity based on the DICOM standard," Computerized Medical Imaging and Graphics, 21(4), 1997, pp.209-218.
[28] D. Dackiewics, et al., "Impact of digital radiography on clinical workflow and patient satisfaction," Journal of Digital Imaging, 13(2), supplement 1, May 2000, pp.200201.
[16] W. Pavlicek, et al. "Quality of Service Improvement from Coupling a Digital Chest Unit with Integrated Speech Recognition, Information, and PACS," Journal of Digital Imaging, 12(4), 1999, pp.191-197.
[29] R.O. Redfern, et al., "Radiology workflow and patient volume: Effect of PACS on technologists and radiologists," Journal of Digital Imaging, 13(2), May 2000, pp.97-100.
[17] J.R. Pilling, "Picture Archiving and Communication Systems: The Users' View," British Journal of Radiology, 76(908), 2003, pp.519-524.
[30] B.I. Reiner, et al., "Impact of filmless radiology on frequency of clinician consultations with radiologists," American Journal of Radiology, 173, November 1999, pp.1169-1172.
[31] G.C. Weatherburn and S. Bryan, "The effect of a PACS on patient radiation doses for examination of the lateral lumbar spine," The British Journal of Radiology, 72, 1999, pp.534-545. [32]
R.O. Mason, "Measuring Information Output: A Communication Systems Approach," Information & Management, 1(5), October 1978, pp.219-234.
[33] T. McGill, V. Hobbs and J. Klobas, "User-Developed Applications and Information Systems Success: A Test of DeLone and McLean’s Model," Information Resources Management Journal, 16(1), 2003, pp.24-45. [34] P.B. Seddon and M.Y. Kiew, “A partial test and development of the DeLone and McLean model of IS success”, Proceedings of the ICIS, 1994, pp.99-110. [35] A. Rai, et al. "Assessing the validity of IS success models: an empirical test and theoretical analysis," Information Systems Research, 13(1), 2002, pp.50-69. [36] M. Igbaria and M. Tan, "The consequences of the IT acceptance on subsequent individual performance," Information & Management, 32(3), 1997, pp.113-121. [37] G. Torkzadeh and W.J. Doll, "The development of a tool for measuring the perceived impact of IT on work," Omega, 27(3), 1999, pp.327-339. [38] Rogers, E.M. Diffusion of Innovations (5th edition), Free Press, New York: NY, 2003.
[39] R.L. Oliver, "A Cognitive Model for the Antecedents and Consequences of Satisfaction," Journal of Marketing Research, 17, 1980, pp.460-469. [40] B.J. Liu, et al. "Trends in PACS image storage and archive," Computerized Medical Imaging and Graphics, 27, 2003, pp.165-174. [41] F. Cao, et al. "Medical Imaging Security in a HIPAA mandated PACS environment," Computerized Medical Imaging and Graphics, 27, 2003, pp.185-196. [42] W.J. Doll and G. Torkzadeh, "The Measure of End-Ùser Computing Satisfaction," MIS Quarterly, 12(2), June 1988, pp.259-274. [43] T.P. Van Dyke, et al. "Measuring IS service quality: concerns on the use of the SERVQUAL questionnaire," MIS Quarterly, 21(2), 1997, pp.195-208. [44] W.J. Kettinger and C.C. Lee, "Perceived service quality and user satisfaction with the IS function," Decision Sciences, 25(5-6), 1995, pp.737-765. [45] L. Raymond, "Organizational characteristics and MIS success in the context of the small business," MIS Quarterly, 8(1), March 1985, pp.37-52. [46] M.J. Tyre and W.J. Orlikowski, "Windows of opportunity: Temporal patterns of technological adaptation in organizations," Organization Science, 5(1), 1994, pp.98118. [47] Nunally, J. Psychometric Methods, 2nd edition, McGrawHill, New York: NY, 1978.
