Towards The Measurement Of Business Intelligence Maturity

4 downloads 23821 Views 482KB Size Report
Jan 7, 2013 - Keywords: Business Intelligence; Maturity Model; Maturity Measurement ... Over the past two decades the importance of business intelligence ...
Association for Information Systems

AIS Electronic Library (AISeL) ECIS 2013 Completed Research

ECIS 2013 Proceedings

7-1-2013

Towards The Measurement Of Business Intelligence Maturity David Raber University of St. Gallen, St. Gallen, St. Gallen, Switzerland, [email protected]

Felix Wortmann University of St. Gallen, St. Gallen, St. Gallen, Switzerland, [email protected]

Robert Winter University of St. Gallen, St. Gallen, St. Gallen, Switzerland, [email protected]

Follow this and additional works at: http://aisel.aisnet.org/ecis2013_cr Recommended Citation Raber, David; Wortmann, Felix; and Winter, Robert, "Towards The Measurement Of Business Intelligence Maturity" (2013). ECIS 2013 Completed Research. Paper 95. http://aisel.aisnet.org/ecis2013_cr/95

This material is brought to you by the ECIS 2013 Proceedings at AIS Electronic Library (AISeL). It has been accepted for inclusion in ECIS 2013 Completed Research by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contact [email protected].

Proceedings of the 21st European Conference on Information Systems

TOWARDS THE MEASUREMENT OF BUSINESS INTELLIGENCE MATURITY Raber, David, University of St. Gallen, Müller-Friedberg-Strasse 8, 9000 St. Gallen, Switzerland, [email protected] Wortmann, Felix, University of St. Gallen, Müller-Friedberg-Strasse 8, 9000 St. Gallen, Switzerland, [email protected] Winter, Robert, University of St. Gallen, Müller-Friedberg-Strasse 8, 9000 St. Gallen, Switzerland, [email protected]

Abstract For the systematic evolution of interdisciplinary socio-technical systems, such as business intelligence (BI), artifacts are needed that comprehensively address multifaceted challenges. To support these strategic improvement initiatives, we proposed a BI maturity model (MM) in prior research. In this paper, we develop an approach to measure maturity of BI in organizations, thereby operationalizing our existing BI MM. In fact, a new maturity measurement instrument is developed that can be used in empirical research. According to the proposed approach, BI maturity can be calculated on the basis of 25 items and organizations are classified into five maturity levels. An exemplary application, using data from 92 organizations, shows that most organizations reside on maturity level three. Furthermore, we investigate the relationship between BI maturity and business benefits of BI. Findings showed that mature organizations have achieved significantly higher business benefits from BI than organizations on lower maturity levels. Keywords: Business Intelligence; Maturity Model; Maturity Measurement

1

Proceedings of the 21st European Conference on Information Systems

1

Introduction

Over the past two decades the importance of business intelligence (BI) has been ever increasing in academia and practice. Information technology (IT) innovations like data warehouse systems and analytical front-end tools have allowed BI to develop into an essential component of information systems (IS) in organizations whose contribution to overall organizational success is undisputed (Davenport et al., 2010; Wixom and Watson, 2010). BI is a “broad category of technologies, applications, and processes for gathering, storing, accessing, and analyzing data to help its users make better decisions” Over time, the role of BI has changed from a „single analytical application‟ view to an organizational capability of strategic importance (Negash and Gray, 2008). Technological challenges are more and more accompanied by questions of organizational implementation of an enterprise capability (e.g. crafting of an enterprise wide BI strategy), IT/business alignment, as well as competence in usage, operations, and further development of a broad solution architecture (Richardson and Bitterer, 2010; Williams and Williams, 2007). Having been denominated as a top technology priority by chief information officers for several years, BI has recently been named as a top business priority, too (Luftman and Ben-Zvi, 2010; McDonald, 2010; Richardson and Bitterer, 2010). However, despite its widely acknowledged importance, putting BI into place still remains challenging (Luftman and Ben-Zvi, 2010) – both from a technological and an organizational perspective. Senior executives and managers need guidance about the pathways of organizational innovation and success through IS (ICIS, 2012). In order to address this challenge, maturity models (MMs) have been proposed as a viable instrument. MMs outline anticipated, typical, logical, and desired evolution paths from an initial to a desired target stage (Kazanjian and Drazin, 1989). Due to their distinctive nature, well-defined MMs are capable of integrating diverse dimensions for measuring, designing and controlling complex, multi-faceted artifacts. Such diverse dimensions covering a wide range of business, technical as well as people-related aspects need to be considered in IS design and IS management on a regular basis. In fact, MMs provide a consistent design and management view on the subject at hand. Over time, MMs have become an established means to identify and explore the strengths and weaknesses of organizations as a whole (e.g. Benbasat et al., 1980; Galbraith, 1982; Kazanjian and Drazin, 1989) or certain domains thereof (e.g. software development (Paulk et al., 1993; Ramasubbu et al., 2008)). While various models for assessing BI maturity were developed during the last years (e.g. Eckerson, 2009; Sacu and Spruit, 2010; Watson et al., 2001), none of these have been operationalized in order to be used in empirically grounded research (Lahrmann et al., 2010). Reliable instruments for measuring the BI maturity level of organizations according to the various models are therefore not available. In this paper, we will address this research gap and develop an instrument for measuring BI maturity. Deriving this instrument is part of a continuous research endeavor in which we intend to explore different aspects of BI MMs from theoretical foundation over construction until application and evaluation. Thus, we base our instrument on a BI MM, which we developed in prior research. More precisely, we focus in this paper on operationalizing the existing BI MM in a transparent way to provide an instrument for empirical research. Thereby, we address the question: how can BI maturity of an organization be measured as a basis for further empirical research? In an exemplary empirical application, we analyze the relationship between BI maturity and business benefits of BI using the proposed measurement instrument. However, validating the instrument in detail and applying it to more complex scenarios is subject to future research. The remainder of this paper is structured as follows. In section two, existing BI MMs are analyzed and our BI MM from prior research is briefly described. We then explicate the approach to operationalize our BI MM in order to measure BI maturity. Afterwards, an example of how our instrument could be employed for empirical analyses is provided before the paper concludes with implications and future work. 2

Proceedings of the 21st European Conference on Information Systems

2

Related Work

MMs – or maturity assessment models – are a widely accepted instrument for systematically documenting and guiding the development and transformation of organizations on the basis of best or common practices (Paulk et al., 1993). The concept of MMs has initially been proposed during the 1970s (Gibson and Nolan, 1974). Driven by the success of prominent examples (e.g. (Ahern et al., 2003; Crawford, 2006; Humphrey, 1988)), numerous MMs have been developed by academics as well as practitioners since then. In the field of IS, a huge number of MM instantiations has been published (Poeppelbuss et al., 2011). A MM typically consists of a sequence of maturity levels for a class of objects (Becker et al., 2009; Klimko, 2001). Each level requires the objects on that level to achieve certain requirements. Maturity in this context is understood as a „measure to evaluate the capabilities of an organization‟ (de Bruin et al., 2005), while the term capability is understood as the ability to achieve a predefined goal (van Steenbergen et al.). With increasing popularity of MMs, criticism addressed a certain arbitrariness and fuzziness of the MM development and design process (Becker et al., 2009; Mettler and Rohner, 2009). In order to address this drawback, de Bruin et al. proposed a MM lifecycle model that is comprised of a scope, design, populate, test, deploy, and maintain phase (de Bruin et al., 2005). Regarding the design phase, two different approaches exist. Whereas in the top-down approach levels are defined first and afterwards characteristics that describe the different dimensions are derived, the bottom-up approach first derives dimensions and characteristics which are afterwards assigned to maturity levels. In the field of BI, various MMs have been proposed (Lahrmann et al., 2010; Wixom and Watson, 2010). In a recent literature review, ten BI MMs were identified and analyzed with respect to methodology and content (Lahrmann et al., 2010). Table 1 presents a brief overview of these models. We updated Lahrmann et al.‟s analysis by one revised model and three recently developed models. Most of these MMs have their origin in practice and are hardly documented. Furthermore, none of the MMs has been subject to a thorough evaluation. Moreover, the respective construction processes have not been published. No. 1 2 3 4 5 6 7 8 9 10 11 12 13

Name (year(s)) Watson et al. (2001) SAS (2004, 2009) Eckerson (2004, 2009) SMC (2004, 2009) Cates et al. (2005) Dataflux (2005) Sen et al. (2006, 2011) HP (2007, 2009) Gartner (2008) Teradata (2008) BIDM (2010) EBIMM (2010) Lukman et al. (2011)

Source (Watson et al., 2001) (Hatcher and Prentice, 2004; Sas Institute, 2009) (Eckerson, 2004; Eckerson, 2009) (Chamoni and Gluchowski, 2004; Schulze et al., 2009) (Cates et al., 2005) (Dataflux, 2005) (Sen et al., 2011; Sen et al., 2006) (Henschen, 2007; Hewlett, 2009) (Rayner and Schlegel, 2008) (Töpfer, 2008) (Sacu and Spruit, 2010) (Chuah, 2010) (Lukman et al., 2011)

Origin Academia Practice Practice Practice Academia Practice Academia Practice Practice Practice Academia Academia Academia

Table 1. Overview of Existing BI Maturity Models Not only the construction process, but also the underlying BI maturity concept should be explicated. The maturity concept outlines what exactly is measured and what the MM‟s purpose is. Also remarkable in this context is the fact, that only one out of the 13 analyzed BI MMs features a theoretical foundation, i.e. only one model is explicitly based on (kernel) theories (Biberoglu and Haddad, 2002): in their stage model for data warehousing, Watson et al. (2001) refer to the stages of growth approach (Gibson and Nolan, 1974). As in this case, an explicated theoretical foundation helps to understand how the different concepts of an MM influence each other. As the analysis of Lahrmann et al. further shows, comprehensiveness of existing BI MMs seems to be an issue, too. Traditional IT

3

Proceedings of the 21st European Conference on Information Systems

topics, e.g. applications, data, and infrastructure are highly present whereas topics as BI organization and BI strategy are widely neglected. This contrasts current IS literature where these two topics gain high visibility, e.g. (Boyer et al., 2010; Vierkorn and Friedrich, 2008). To address the aforementioned shortcomings of existing MMs in the field of BI we constructed a BI MM in prior research, which (a) comprehensively conceptualizes BI, (b) is developed in a transparent way based on an explicit maturity concept and (c) is informed by theory. For detailed information on our BI MM and its construction the reader can refer to (Raber et al., 2012). In summary, our BI MM is comprised of five levels and the five maturity dimensions strategy, organization, information technology (IT), quality and use. In the following, the five levels of our BI MM are briefly described. Level one of the BI MM is characterized by a high degree of decentralism with almost no standardization efforts, representing an early and immature state of BI. Organizations that achieve level two are clearly oriented towards centrally managed BI in terms of governance and organizational setup. Level three of the BI MM, represents the final step towards centralization and integration, as well as an intermediate stage with respect to optimization. On level four, organizations are realizing the full potential of BI and drive advanced strategic topics such as BI portfolio management and business cases for BI. For achieving the highest level five of BI maturity, a sustainable and continuous management of BI needs to be established. In terms of capabilities, this stage of maturity requires a comprehensive BI strategy to be specified and regularly updated. In addition, BI performance management and pro-active data quality management need to be fully deployed. In the paper at hand, we develop a measurement instrument for this BI MM.

3

Operationalizing BI Maturity for Empirical Research

As a measure for BI maturity, the most straightforward candidate is the level an organization has reached in the BI MM. For assessing which of the five maturity levels is reached by an organization, we built upon an approach used for classifying companies into their corresponding business and IS strategy types (Sabherwal and Chan, 2001), which was refined in the field of service oriented architectures (Joachim et al., 2011). First of all, we create a set of 25 items as a basis to measure BI maturity across the five maturity levels. Furthermore, using ideal maturity profiles, we calculate the distance of an organization to each maturity level by applying the Euclidean metric in two steps. The maturity level having the smallest Euclidean distance represents the overall BI maturity level of the organization. This approach is summarized in Figure 1 and presented in detail in the following. The goal of phase A of our analytical approach is to develop a survey instrument that is based on essential characteristics of the existing BI MM (cf. (Raber et al., 2012)). BI capabilities included in the existing BI MM form several item groups of progressively more difficult indicators of BI maturity. However, the approach we describe in the following requires these capability groups to be summarized into a single item to enable measurement. Therefore, capabilities of the five maturity dimensions (i.e. strategy, organization, IT, quality, and use) have been analyzed and condensed resulting in a total of 25 items. In principal, the challenge was to derive a limited set of coherent items from the BI MM, which summarize all aspects of the original capabilities of the BI MM and could measure BI maturity. Thus this approach was also aimed at balancing effort to measure, i.e. number of items and comprehensiveness of the measurement instrument. After creation, the list of items was structured into a survey using a five-point Likert scale from (1) “strongly disagree” to (5) “strongly agree”. This survey instrument represents the initial version of our BI maturity measurement instrument and is shown in the Appendix. Thorough evaluation and validation of the instrument is subject to future research and not in scope of the paper at hand presenting the development and a first application of the instrument. In order to measure the survey responses against the maturity levels of the MM, ideal maturity profiles, i.e. characteristic values, were defined for each maturity level. We thereby follow a theoretical approach developed by Sabherwal & Chan (2001), which was later adapted by Joachim et al. (2011). These characteristic values are based on the assumption that BI maturity increases in a

4

Proceedings of the 21st European Conference on Information Systems

linear manner in equidistant steps and on the fact that items are measured using a five-point Likert scale. Thus characteristic values vli for each item i and all levels l are defined as follows: v1i=1 (Level 1), v2i=2 (Level 2), v3i=3 (Level 3), v4i=4 (Level 4), v5i=5 (Level 5). The ideal maturity profile for level one for example is then represented by all 25 items having a rating of one. A.1: Develop set of items and a scale to be used as survey instrument.

B.1: Calculate distances to each maturity level (scope: maturity dimension)

C: Application of BI maturity instrument in empirical research

 Items are derived from our BI MM

 For an organization, Euclidean distances are calculated between each maturity dimension defined in the BI MM and the characteristic values of the items for each maturity level

 The calculated BI maturity level can be used for various empirical research methods, e.g. cluster analysis, regressions, structural equation models (SEM)

A.2: Define characteristic values for each maturity level  Profiles of characteristic values have to be defined for the items and each maturity level  Equidistant steps from one to five are used to reflect the maturity levels

B.2: Classify maturity dimensions  Each maturity dimension of an organization is classified into the maturity level with least distance

B.3: Calculate distances to each maturity level (scope organization)  Euclidean distances are calculated between an organization and the characteristic values for each maturity level  Basis for calculation are the classified maturity dimensions

B.4: Classify organizations  Each organization is classified into the maturity level with least distance Phase A: Development of survey instrument and characteristic values

Phase B: Classification of each organization into a BI maturity level

Phase C: Exemplary empirical application of BI maturity instrument

Figure 1. Analytical Approach In the second phase, a twofold application of the Euclidean metric is utilized and yields the BI maturity level of an organization as a result. While the squared statistical distance used by Joachim et al. (2011) provides greater flexibility in regards to item scales, there is no need for us to build upon a sophisticated metric as all of our items are measured on a five-point Likert scale. To measure BI maturity, the Euclidean distance is computed for the specific BI maturity dimension d of an organization o between the answers xoi given to the specific items i belonging to the set of items of this dimension Id, and their defined characteristic values for the specific maturity level vli. This results in a total of five distance values DistDod for each organization o per maturity level:

DistDod (l ) 

 x

iI d

 vli  for 1  d  nd and 1  l  nl 2

oi

With the total number of dimensions nd and the total number of levels nl, i.e. nd=5 and nl=5 in the case of our BI MM. Next, each BI maturity dimension of an organization can be classified into one of the five maturity levels by using these distance values. Simply the level with the least distance is attributed to every maturity dimension yielding five integer values for each organization, which characterize the organization‟s BI maturity:

LevelDod  m, such that DistDod m  min ( DistDod (l )) for 1  m  nl 1l nl

5

Proceedings of the 21st European Conference on Information Systems

To compute the overall BI maturity level LevelOo for an organization o, the Euclidean distance is again applied to calculate the distance between the five maturity values LevelDod of each dimension, and the characteristic value of the specific maturity dimension d for each level l, uld, which is simply l. Again, the least distance of the five resulting distance values DistOo per organization determines the organization‟s respective BI maturity level:

Let DistOo l  

nd

 LevelD d 1

od

 uld  for 1  l  nl , then 2

LevelOo  m, such that DistOo m   min DistOo l  for 1  m  nl 1l nl

Phase C of our analytical approach finally yields an exemplary empirical application of the developed BI maturity instrument.

4

Exemplary Empirical Application of the BI Maturity Instrument

4.1

Data Collection

In this section an exemplary empirical application of the BI maturity instrument is presented. Data was collected using a paper questionnaire distributed at a BI practitioner event and an online version of the questionnaire. It was ensured that participant segments did not overlap and participants were provided with a short introduction to the subject of the questionnaire. The paper questionnaire was returned by 44 out of 89 participants of the conference, yielding a response rate of 49.4%. The conference was attended by BI/data warehousing specialists and executives working in business, management, and IT functions. In addition, the online questionnaire was sent to 78 practitioners who attended the conference in previous years and was completed by 48 recipients resulting in a response rate of 61.5%. Table 2 summarizes the characteristics of the overall sample. Industry sector Automotive industry Services Financial services Public administrations IT and communications Wholesale & retail Other industries Sum

No. % 11 12.0 20 21.7 17 18.5 4 4.3 15 16.3 7 7.6 18 19.6 92 100

Employees 1-250 251-1000 1001-5.000 5.001-10.000 > 10.000 Not available Sum

No. % 20 21.7 9 9.8 16 17.4 10 10.9 35 38.0 2 2.2 92 100

Table 2 Sample Characteristics

4.2

Data Analysis and Results

Using this data, we applied phase B of our analytical approach (cf. Figure 1), i.e. the measurement algorithm, in order to compute the BI maturity level of each organization. To determine the quality of the measurement instrument, reliability was assessed on the basis of Cronbach‟s alpha. The Cronbach alpha coefficient computed for the instrument was .905. Therefore, according to established research practice reliability of the instrument can be judged as „excellent‟ (Kline, 1999). Table 3 provides an overview of the results and classifies the organizations according to their BI maturity level grouped by industry. 60% of the overall respondent organizations have been classified into maturity level three, and around 85% of the organizations have already reached level three or more. Face validity of these results is given. By today, BI is a mature domain: indeed, 70% of participating organizations indicated that they had been using BI for more than five years at the time of the survey. This correlates with the fact that 1% of organizations are on maturity level one and 14% on maturity level two. The IT and

6

Proceedings of the 21st European Conference on Information Systems

communications sector appears to be the most mature industry in our sample. Organizations from this sector have reached at least BI maturity level three and the IT and communications industry also has the only organizations that reached the highest level.

Level 1 Level 2 Level 3 Level 4 Level 5 Total

Automotive Services Financial Public Admi- IT and com- Wholesale Other Total industry services nistrations munications & retail industries 0% 5% 0% 0% 0% 0% 0% 1 18% 25% 12% 50% 0% 0% 11% 13 45% 55% 71% 50% 40% 57% 83% 55 36% 15% 18% 0% 47% 43% 6% 21 0% 0% 0% 0% 13% 0% 0% 2 11 20 17 4 15 7 18 92

Table 3 BI Maturity Level of Organizations Grouped by Industry Sector With the exemplary empirical application of our BI maturity instrument we intend to answer the question: “How is BI maturity related to business benefits?” We assume that organizations at a high level of BI maturity are able to generate greater business benefits than organizations at a lower level of BI maturity. In order to investigate these relationships, we construct a structural equation model. The model relates the output of our BI maturity instrument, i.e. the BI maturity level, as an independent variable and three different business benefits of BI that we derived from literature, as dependent variables. In fact, we take the instrument for assessing business value of BI proposed by Elbashir et al. (2008) as a basis for our exemplary research model. The model of Elbashir et al. was slightly adapted to overcome the fact that some characteristics are specific to the manufacturing industry (e.g. reduced inventory levels). According to the instrument of Elbashir et al. (2008), benefits of BI at the business process level can be measured on the basis of three key concepts: (1) internal process efficiency benefits refer to benefits that arise from improvement in the efficiency of internal processes, (2) business supplier/partner relation benefits include benefits that organizations gain from improved relations with their business partners and suppliers, and (3) customer intelligence benefits that arise from a better understanding of the customer and the market. These three concepts were also assessed in our questionnaire. In order to validate the model we employed Partial Least Square (PLS) analysis using the SmartPLS software (version 2) (Ringle et al., 2005). PLS is a regression-based technique that allows for estimating and testing relationships between constructs (Chin, 1998). The PLS technique was chosen to conduct the analyses due to its ability to handle a wide range of sample sizes and constructs with fewer items (Hair et al., 2009).

Construct BI maturity

Label

BIM IB1 Internal process IB2 efficiency IB3 benefits IB4 SB1 Business supplier/partner SB2 relation benefits SB3 CB1 CB2 Customer intelligence CB3 benefits CB4 CB5

Indicator (five-point Likert scale from "strongly disagree" to "strongly agree") BI maturity level based on computations from step 2 (cf. Figure 1) Improved efficiency of internal processes Increased staff productivity Reduction in the cost of effective decision-making Reduced operational costs Reduction in the cost of transactions with business partners/suppliers Improved coordination with business suppliers/partners Increased responsiveness to/from suppliers Increased revenues Reduction of lost sales Increased geographic distribution of sale Reduced marketing costs Reduced time-to-market products/services

Loadings 1.000 0.816 0.845 0.747 0.708 0.767 0.946 0.931 0.798 0.833 0.840 0.681 0.729

Table 4. Measurement Instrument for Business Benefits of BI

7

Proceedings of the 21st European Conference on Information Systems

The corresponding items to measure business benefits of BI are listed in Table 4. In accordance to the original model from Elbashir et al., our model was specified as a reflective model following the criteria of Jarvis et al. (2003). As the main contribution of this paper is developing a BI maturity measurement instrument and showing its general applicability, the quality criteria of the exemplary application are not described in detail here. However, reliability and validity criteria have been met: Cronbach‟s α exceeds 0.79 for every latent variable and all composite reliability values are higher than 0.86, which clearly exceeds the requested value of 0.7 (Nunnally and Bernstein, 1994), thus proving internal consistency reliability of our model. Furthermore, the reliabilities of the indicators (i.e.item loadings) fulfil the demanded 0.7 (Nunnally and Bernstein, 1994) in all but one case (cf. Table 4). However, the loading of CB4 (0.681) is still sufficiently larger than 0.6 (Bradley et al., 2006; Hair Jr et al., 1998). Our model exhibits an average variance extracted (AVE) for every latent variable of at least 0.6, which satisfies the 0.5 threshold (Chin, 1998). Together with the item loadings, this establishes convergent validity of the research model. As each AVE value is higher than the squared correlations with all other latent variables, the Fornell-Larcker criterion is also met showing discriminant validity (Fornell and Larcker, 1981). The cross-loadings support this observation, as for every indicator the correlation with its respective latent variable is significantly higher than with any other latent variable (Chin, 1998; Götz et al., 2010). Having established measurement validity and reliability, the next step is to test the structural model for the hypothesized paths. Primary evaluation criteria for this purpose are the R2 measures as well as level and significance of the path coefficients (Hair Jr et al., 2011). R2 values indicate the amount of variance of a dependent variable explained by the model (Chin, 2010). Path coefficients indicate the strengths of the relationships between the dependent and independent variables. Following Chin (2010), we performed the bootstrapping sampling method to generate 500 samples to estimate path coefficient‟s significance. Figure 2 depicts the results of our exemplary research model, which indicates positive and significant relationships between our BI maturity construct and the three business benefits of BI. The research model explained 30.2% of the variance in the dependent variable “internal process efficiency”, 19.5% in “business supplier/partner relations, as well as 18.2% in customer intelligence. The R2 values indicate that BI maturity is only one of many organizational factors influencing the three aspects of business benefits included in our model. Business Benefits of BI 0.549*

BI Maturity

0.441*

0.427*

Internal Process Efficiency (R2=0.302) Business Supplier/Partner Relations (R2=0.195) Customer Intelligence (R2=0.182)

* Indicates that the coefficient is significant at p < 0.001

Figure 2. Exemplary Empirical Application of BI Maturity in SEM

5

Conclusion and Future Research

By developing an instrument to measure the BI maturity of organizations, we created a construct that can be used in future empirically driven research on the business value of BI. For this purpose, we operationalized an existing BI MM by means of a survey instrument comprising 25 items (cf. Appendix). These 25 items enable us to calculate the BI maturity level of an organization with respect to the five BI MM levels. Thus, this BI maturity measurement instrument can be easily used in future empirical and survey driven studies. In order to demonstrate the applicability and usefulness of our instrument, we documented an exemplary application in this paper. The calculated BI maturity level was used as an independent

8

Proceedings of the 21st European Conference on Information Systems

variable in a structural equation model to explain business benefits as dependent variables. The quality criteria in PLS analysis were met and the result shows a positive and significant relation between BI maturity and achieved business benefits of BI. This can be seen as a first proof for the BI maturity measurement instrument. However, validity and reliability of our instrument have yet to be confirmed in more elaborate and more detailed analyses. For future research, a larger number of organizations need to be analyzed. In addition, the model of Elbashir et al. has its own limitations: The study relies on subjective “perception-based” measures and the underlying sample is comprised of customers of only one BI software vendor affecting external validity. Following existing research approaches, we build upon the assumption that BI maturity increases in equidistant steps. It should be examined if there are any specifics inherent to BI maturity that challenge this assumption. A further limitation is that the data could potentially be subject to common method bias, which may occur when data are collected via only one method (Campbell and Fiske, 1959). In the future, we plan to use qualitative interviews to assess the BI maturity level of some of the organizations, which have participated in our survey, in order to confirm the results of the proposed maturity measurement instrument. Comparing the results from our instrument and the outcomes of the qualitative interviews can eventually lead to adjustments of the instrument. Furthermore, quantitative validation can complement the depicted qualitative validation. Finally, this instrument represents one part of our on-going research process around the topic of BI MM. It extends and complements our existing BI MM and we intend to use our instrument in order to validate the BI MM itself. Indeed, the measurement instrument provides a basis for validation, because it “applies” the MM to organizations to calculate their as-is position with respect to the MM. How this as-is position, i.e. the BI maturity level, can then be used to evaluate the BI MM is subject to future research. The implications for practice are manifold. First of all, our work provides a basis for determining the level of BI maturity in a systematic, transparent and grounded way. Our exemplary empirical application indicates that BI maturity is a driver for business benefits. Therefore, regularly applying our instrument as a basis for continuous benefit improvement is a viable practice for managers in charge of BI. Second, our approach of measuring maturity can be leveraged by professionals from various domains to overcome methodological weaknesses of the MMs they apply. Since our approach is not BI specific, it can be used for other domains where MMs are suitable design or management instruments, e.g. because the phenomenon at hand is complex and multi-faceted. Third, the survey results can be leveraged by individual organizations for a rudimentary benchmark. Organizations often lack a reliable basis or “reference” for assessing their performance. Table 3 depicting the maturity levels across industries can serve as a starting point to create such a basis for performance assessment.

References Ahern, D. M., Clouse, A. and Turner, R. (2003). CMMI Distilled: A Practical Introduction to Integrated Process Improvement, Addison-Wesley, Boston. Becker, J., Knackstedt, R. and Pöppelbuß, J. (2009). Developing Maturity Models for IT Management - A Procedure Model and its Application, Business & Information Systems Engineering, 1 (3), pp. 213-222. Benbasat, I., Dexter, A. S. and Mantha, R. W. (1980). Impact of Organizational Maturity on Information System Skill Needs, MIS Quarterly, 4 (1), pp. 21-34. Biberoglu, E. and Haddad, H. (2002). A Survey of Industrial Experiences with CMM and the Teaching of CMM Practices, Journal of Computing Sciences in Colleges, 18 (2), pp. 143-152. Boyer, J., Frank, B., Green, B., Harris, T. and Van De Vanter, K. (2010). Business Intelligence Strategy - A Practical Guide for Achieving BI Excellence, MC Press Online, Ketchum, ID. Bradley, R. V., Pridmore, J. L. and Byrd, T. A. (2006). Information Systems Success in the Context of Different Corporate Cultural Types: An Empirical Investigation, Journal Of Management Information Systems, 23 (2), pp. 267-294. Campbell, D. T. and Fiske, D. W. (1959). Convergent and discriminant validation by the multitraitmultimethod matrix, Psychological Bulletin, 56 (2), pp. 81-105.

9

Proceedings of the 21st European Conference on Information Systems

Cates, J. E., Gill, S. S. and Zeituny, N. (2005). The Ladder of Business Intelligence (LOBI): a framework for enterprise IT planning and architecture, International Journal of Business Information Systems, 1 (1-2), pp. 220-238. Chamoni, P. and Gluchowski, P. (2004). Integrationstrends bei Business-Intelligence-Systemen Empirische Untersuchung auf Basis des Business Intelligence Maturity Model, Wirtschaftsinformatik, 46 (2), pp. 119-128. Chin, W. W. (1998). The Partial Least Squares Approach to Structural Equation Modeling, In Modern Methods for Business Research(Ed, Marcoulides, G. A.) Lawrence Erlbaum Associates, Mahwah, pp. 295-336. Chin, W. W. (2010). How to Write Up and Report PLS Analyses, In Handbook of Partial Least Squares. Concepts, Methods and Applications(Eds, Vinzi, V. E., Chin, W. W., Henseler, J. and Wang, H.) Springer, Berlin, Heidelberg, pp. 655-690. Chuah, M.-H. (2010). An enterprise business intelligence maturity model (EBIMM): Conceptual framework, Fifth International Conference on Digital Information ManagementIEEE, pp. 303-308. Crawford, J. K. (2006). The Project Management Maturity Model, Information Systems Management, 23 (4), pp. 50-58. Dataflux (2005). Enterprise Data Management Maturity Model, Dataflux. Davenport, T. H., Harris, J. G. and Morison, R. (2010). Analytics at Work: Smarter Decisions, Better Results, Harvard Business Press, Boston. de Bruin, T., Rosemann, M., Freeze, R. and Kulkarni, U. (2005). Understanding the Main Phases of Developing a Maturity Assessment Model, 16th Australasian Conference on Information Systems (ACIS 2005) (Eds, Campbell, B., Underwood, J. and Bunker, D.) University of Technology Sydney, Sydney, Australia, 30.11.2005, pp. 1-10. Eckerson, W. W. (2004). Gauge Your Data Warehouse Maturity, DM Review, 14 (11), pp. 34. Eckerson, W. W. (2009). TDWI's Business Intelligence Maturity Model, Chatsworth. Elbashir, M. Z., Collier, P. A. and Davern, M. J. (2008). Measuring the effects of business intelligence systems: The relationship between business process and organizational performance, International Journal of Accounting Information Systems, 9 (3), pp. 135–153. Fornell, C. and Larcker, D. (1981). Evaluating structural equation models with unobservable variables and measurement error, Journal of Marketing Research, 18 (1), pp. 39-50. Galbraith, J. (1982). The stages of growth, Journal of Business Strategy, 3 (1), pp. 70-79. Gibson, C. F. and Nolan, R. L. (1974). Managing the four stages of EDP growth, Harvard Business Review, 52 (1), pp. 76-88. Götz, O., Liehr-Gobbers, K. and Krafft, M. (2010). Evaluation of Structural Equation Models Using the Partial Least Squares (PLS) Approach, In Handbook of Partial Least Squares(Eds, Vinzi, V. E., Chin, W. W., Henseler, J. and Wang, H.) Springer, Berlin Heidelberg, pp. 691-711. Hair, J. F., Black, W. C., Babin, B. J. and Anderson, R. E. (2009). Multivariate Data Analysis, Prentice Hall, Upper Saddle River, NJ. Hair Jr, J. F., Anderson, R. E., Tatham, R. L. and Black, W. C. (1998). Multivariate Data Analysis, Prentice Hall, Upper Saddle River (New Jersey). Hair Jr, J. F., Ringle, C. M. and Sarstedt, M. (2011). PLS-SEM: Indeed a Silver Bullet, Journal of Marketing Theory and Practice, 19 (2), pp. 139-151. Hatcher, D. and Prentice, B. (2004). The Evolution of Information Management, Business Intelligence Journal, 9 (2), pp. 49-56. Henschen, D. (2007). HP Touts Neoview Win, Banking Solution, BI Maturity Model, Intelligent Enterprise, 10 (10), pp. 9. Hewlett, P. (2009). The HP Business Intelligence Maturity Model: Describing the BI journey, Hewlett-Packard Development Company L.P. Humphrey, W. S. (1988). Characterizing the Software Process: A Maturity Framework, IEEE Software, 5 (2), pp. 73-79. ICIS (2012). ICIS 2012 Track Description: Economics and Value of IS, Vol. 2012.

10

Proceedings of the 21st European Conference on Information Systems

Jarvis, C. B., MacKenzie, S. B. and Podsakoff, P. M. (2003). A Critical Review of Construct Indicators and Measurement Model Misspecification in Marketing and Consumer Research, Journal of Consumer Research, 30 (2), pp. 199-218. Joachim, N., Beimborn, D. and Weitzel, T. (2011). An Instrument for Measuring SOA Maturity, Icis 2011. Kazanjian, R. K. and Drazin, R. (1989). An empirical test of a stage of growth progression model, Management Science, 35 (12), pp. 1489-1503. Klimko, G. (2001). Knowledge Management and Maturity Models: Building Common Understanding, Bled, Slovenien, pp. 269–278. Kline, P. (1999). Handbook of Psychological Testing, Routledge, London. Lahrmann, G., Marx, F., Winter, R. and Wortmann, F. (2010). Business Intelligence Maturity Models: An Overview, VII Conference of the Italian Chapter of AIS (itAIS 2010)(Eds, D'Atri, A., Ferrara, M., George, J. F. and Spagnoletti, P.) Italian Chapter of AIS, Naples, Italy. Luftman, J. N. and Ben-Zvi, T. (2010). Key Issues for IT Executives 2009: Difficult Economy's Impact on IT, MIS Quarterly Executive, 9 (1), pp. 49-59. Lukman, T., Hackney, R., Popovic, A., Jaklic, J. and Irani, Z. (2011). Business Intelligence Maturity: The Economic Transitional Context Within Slovenia, Information Systems Management, 28 (3), pp. 211-222. McDonald, M. (2010). Without the Business in Business Intelligence, BI Is Dead! Mettler, T. and Rohner, P. (2009). Situational Maturity Models as Instrumental Artifacts for Organizational Design, 4th International Conference on Design Science Research in Information Systems and Technology (DESRIST 2009) Association for Computing Machinery, Philadelphia, PA, 07.05.2009, pp. 9. Negash, S. and Gray, P. (2008). Business Intelligence, In Handbook on Decision Support Systems 2(Eds, Burstein, F. and Holsapple, C. W.) Springer, Berlin, Heidelberg, pp. 175-193. Nunnally, J. C. and Bernstein, I. H. (1994). Psychometric theory, McGraw-Hill, New York. Paulk, M. C., Curtis, B., Chrissis, M. B. and Weber, C. V. (1993). Capability Maturity Model, Version 1.1, IEEE Software, 10 (4), pp. 18-27. Poeppelbuss, J., Niehaves, B., Simons, A. and and Becker, J. (2011). Maturity Models in Information Systems Research: Literature Search and Analysis, Communications Of The Association For Information Systems, 29 (1), pp. 505-532. Raber, D., Winter, R. and Wortmann, F. (2012). Using Quantitative Analyses to Construct a Capability Maturity Model for Business Intelligence. Ramasubbu, N., Mithas, S., Krishnan, M. S. and Kemerer, C. F. (2008). Work Dispersion, Processbased Learning, and Offshore Software Development Performance, MIS Quarterly, 32 (2), pp. 437458. Rayner, N. and Schlegel, K. (2008). Maturity Model Overview for Business Intelligence and Performance Management, Gartner, Stamford. Richardson, J. and Bitterer, A. (2010). Findings: The Risks of Losing Faith in BI, Gartner, Stamford. Ringle, C., Wende, S. and Will, A. (2005). SmartPLS 2.0, University of Hamburg, Hamburg. Sabherwal, R. and Chan, Y. E. (2001). Alignment Between Business and IS Strategies: A Study of Prospectors, Analyzers, and Defenders, Information Systems Research, 12 (1), pp. 11-33. Sacu, C. and Spruit, M. (2010). BIDM - The Business Intelligence Development Model, 12th International Conference on Enterprise Information SystemsSciTePress. Sas Institute, I. (2009). Information Evolution Model, SAS Institute Inc. Schulze, K.-D., Besbak, U., Dinter, B., Overmeyer, A., Schulz-Sacharow, C. and Stenzel, E. (2009). Business Intelligence-Studie 2009, Steria Mummert Consulting AG, Hamburg. Sen, A., Ramamurthy, K. and Sinha, A. P. (2011). A Model of Data Warehousing Process Maturity, IEEE Transactions On Software Engineering, 99. Sen, A., Sinha, A. P. and Ramamurthy, K. (2006). Data Warehousing Process Maturity: An Exploratory Study of Factors Influencing User Perceptions, IEEE Transactions on Engineering Management, 53 (3), pp. 440-455.

11

Proceedings of the 21st European Conference on Information Systems

Töpfer, J. (2008). Active Enterprise Intelligence, In Active Enterprise Intelligence(Eds, Töpfer, J. and Winter, R.) Springer, Berlin, Heidelberg, pp. 1-28. van Steenbergen, M., Bos, R., Brinkkemper, S., van de Weerd, I. and and Bekkers, W. (2010). The design of focus area maturity models, Vol. 6105 St. Gallen, Switzerland, pp. 317-332. Vierkorn, S. and Friedrich, D. (2008). Organization of Business Intelligence, BARC Institute, Würzburg. Watson, H. J., Ariyachandra, T. R. and Matyska Jr, R. J. (2001). Data Warehousing Stages of Growth, Information Systems Management, 18 (3), pp. 42-50. Williams, S. and Williams, N. (2007). The profit impact of business intelligence, Morgan Kaufmann, San Francisco, CA. Wixom, B. H. and Watson, H. J. (2010). The BI-Based Organization, International Journal of Business Intelligence Research, 1 (1), pp. 13-28.

Appendix: BI Maturity Measurement Instrument Quality

IT

Org.

Use

Strategy

Maturity Dimensions

Item (five-point Likert scale from (1) “strongly disagree” to (5) “strongly agree”)

BI is characterized by: X X X

BI is financially supported/led by influential persons from business Significant BI decisions are made by a BI steering committee within business BI is based on a comprehensive BI strategy that is regularly updated BI management is based on elaborated methods such as cost accounting, balanced scorecard X or portfolio management IT acts as a business partner and takes an active role in improving business practices on the X basis of BI X BI organization and responsibilities are centralized inside the enterprise X Development of BI solutions is based on a BI specific standard development process X Agile concepts are used to develop BI solutions X BI applications are operated on the basis of standards such as ITIL X Standard reports and dashboards ensure a high quality information supply Advanced analytical requirements are addressed on the basis of existing OLAP tools and X software for pro-active analyses X Frontends are integrated and enable a seamless access to information X BI backend systems are centralized and standardized X Information is integrated across departmental borders Clearly defined responsibilities, standards and principles do exist in the following areas of BI – average of these five items is used: Tools and applications Business content, i.e. KPIs and dimensions X Management and sourcing processes Development processes Operations processes BI applications are used by the following group of people: X Top management X Middle management X Analysts, data scientists X Operative users Data quality is ensured by the following means: Roles, tasks and responsibilities are clearly defined and document in the context of data X quality X Core business objects, performance indicators and dimensions are consistently defined X Data quality is continuously measured in order to pro-actively manage data quality BI systems have the following properties: X Operation of BI systems is based on defined service-level agreements X State of the art BI frontends are used X Response times of BI systems enable efficient and effective usage

12