Oct 14, 2008 - Complex indexes of autonomy within the COBRA research. ..... expert (group of A) in accordance to relative importance of involved items.
Methodological research note COBRA meeting December 2005
The use of composed indexes in the COBRA-research: potential dangers and options Verhoest Koen & Verschuere Bram
1.
Introduction................................................................................................................................. 2
2.
Steps, methods and tests for index construction ....................................................................... 2
3.
Review of some existing indexes on autonomy ......................................................................... 4
4.
Kind of complex indexes used in the COBRA research............................................................. 8
5.
Complex indexes of autonomy within the COBRA research...................................................... 9 Scope ................................................................................................................................................... 9 Measurement ....................................................................................................................................... 9 Construction and Validation................................................................................................................. 9 Examples of autonomy indexes in survey research .......................................................................... 10 Methodological strengths and weaknesses of our autonomy indexes .............................................. 12 1. Concept and selection of variables ...................................................................................... 12 2. Quantifying values................................................................................................................ 12 3. allocation of weights to variables ......................................................................................... 13 4. Techniques for computing indexes ...................................................................................... 14 5. Checking for sensitivity ........................................................................................................ 14 6. Levels of aggregation........................................................................................................... 14
6.
Conclusions and next steps ..................................................................................................... 14 Limitations ...................................................................................................................................... 14 Opportunities .................................................................................................................................. 15 Reference List ................................................................................................................................ 16
1
1. Introduction In research on autonomy of agencies in general and in the COBRA research in specific, composed indexes are frequently used to measure autonomy and other concepts. However, the construction of complex indexes, which are composed of several variables is not without any risks (bias, miscalculation and misinterpretation). These risks are frequently underestimated or not explicitly acknowledged. The advantage of composed indexes is the aggregation of information, stemming from multiple variables and the possibility to capture the ‘broad picture’. Moreover, indexes on organisational features like management autonomy enable us to rank organizations with respect to that feature. Also we can use such indexes on e.g. management autonomy in explanatory research as dependent or independent variable. Complex indexes can even aggregate multidimensional information (e.g. an overall index on autonomy, aggregating management autonomy and policy autonomy data). However, the method of construction as well as the weighting of the variables has an influence on the composed index. The construction of indexes runs through several phases which allows for a high level of subjectivity and bias. Therefore, indexes should be used with reasonable caution in descriptive and explanatory research. In this short research note, we first highlight some methods of index construction and discuss some potential pitfalls, as well as solutions. Then we discuss some indexes of autonomy used on previous research on autonomy of agencies and their strengths and flaws. In a third part we highlight some different kind of indexes that we have used until now in COBRA research. In a fourth part we focus on the indexes of managerial and policy autonomy and discuss more in depth how we try to validate these indexes and make them robust.
2. Steps, methods and tests for index construction1 In constructing a composed index, several subsequent steps are to be taken 1. choice of concept to be measured, 2. selection of relevant variables 3. judgment of quality of data 4. analysis of relationship between variables 5. quantifying the values on the variables and normalization (if necessary) 6. assigning relative weights to the variables 7. computing the index 8. levels of aggregation 9. checking for sensitivity of method, uncertainty and weights
Each of these steps involve some discretion on the part of the researcher and can cause biases. In the literature several methods for constructing composed indexes are suggested. • Mostly (un)weighted summation is used. Literature concerning evaluation techniques and multicriteria analysis shows that this technique is not neutral. Weighted summation can only be used appropriately when combining quantitative values on the variables with quantitative weights. To use weighted summation correctly the variable or indicator should be both continuous and cardinal, because the technique treats the values and weights as such. However, in studies the variables are mostly ordinal of nature. The same can be said of the weights allocated to the variables which are also ordinal. • When using qualitative values and qualitative weights as a starting point to construct indices, other techniques are more valid, such as the regime method or expected value method (Ministerie van 1
(see e.g. Beuselinck and Notebaert 2003)
2
Financiën, Afdeling Beleidsanalyse en -instrumentatie 1992b). • Making logical – theoretical indexes based on subsequent levels with respect to variables • Inductive-empirical indexes based on factor analysis of the variables • …. The problem of assigning weights to the variables is crucial and different techniques are available (see table 1). In table 1 we make a distinction between allocating relative qualitative weights (i.e. the relative ranking of indicators as being most important, second most important…) and allocating quantitative weights (i.e. allocating numeric values depending on importance, mostly adding up to 1). Method for assigning weights 1. Assigning equal weights to all variables by researcher 2. Assigning weights by reference to theory (Elgie, Leerdam) by researcher 3. Assigning weights by researcher within certain ranking of weights by using methods like direct quantifying, expected value and random weighting 4.
Assigning relative weights by researcher based on multiple regression analysis
5.
Constructing index based on principal component analysis Assigning relative quantitative weights by respondents
6.
7.
8.
Assigning relative qualitative weights by respondents with quantification of weights done by researcher Making weights endogenuous
Advantages and disadvantages
When
Easy and quick, unweighted summation
When variables correlate strongly or not at all
Based in theory but also in reality?
When theory allows for assigning relative weights to variables
When qualitative rankings of variables is known, you can use these methods to calculate the relative quantatitive weight of each variable. Advantage is that these methods do not assume equal distribution of the quantitative weights. Moreover one can compute indexes based on different weighting methods in order to check for weight sensitivity. Problem is how to establish the relative ranking E.g. index on ‘National innovation capacity’ (Porter & Stern, 1999).
When the relative qualitative weights of each variable is known
The factor which account for 80% of variance is used as index for original set of indicators When respondents involved in the organizations that are studied assign themselves the relative weights of variables. Reflects importance of variable in practice. Problem of generalisation to other organisations? Can be done through panels, surveys and interviews Is easier for respondent. Possibility of bias because of quantification of researcher Can be done through panels, surveys and interviews The computer chooses for each case (organization) the best set of weights in order to give the organization the best possible score (Cherchye, Moesen en Van Puyenbroeck 2003)
When relative influence of each variable on variable is known. Only in case where indexes are composed on mutually uncorrelated variables and where there is a predictive model of the influence of variables on the concept measured by the index When multidimensionality of data is limited When the relative ranking of variables is unsure or unknown to the researcher
When the relative ranking of variables is unsure or unknown to the researcher
When the individual conditions for each case have to be taken into account
Table 1. methods for assigning weights
3
With respect to indexes several sensitivity tests are possible: Uncertainty tests for • Used methods for computing the index: based on the same data and variables different methods can be used to compute the index and then compare the resulting values of the index • scores on variables: less relevant, depends on quality of data • weights: within the same qualitative ranking of variables it is possible to assign different quantitative weights using different methods and then compare the resulting values of the index
3. Review of some existing indexes on autonomy In a previous conference paper we hypothesize that the inconclusiveness of contemporary research on the link between organisational autonomy and performance may be due, among others, to the different operationalisation of autonomy and methodological choices in research on autonomy (Verhoest et al. 2003). First, we point at some analytical differences in the concept of autonomy and its measurement that is used. Second, we discuss differences in index construction at the methodological level. For a full discussion about the influence of different data collection strategies and data analysis, please see the original paper (Verhoest et al. 2003). At the analytical level, a first observation is that most of the studies stick to a quite general level of operationalisation of their concept of autonomy, measuring the included dimensions by one or two indicators. Moreover, the indicators used highlight only selective aspects of the included dimensions. This selectivity may be problematic because studies do not always justify the choice of indicators thoroughly, for example by using factor analysis. Therefore, the reader can not be certain that the selected indicators measure the presumed dimension properly. This is certainly the case for the studies using the formal-legal status as the only indicator of autonomy. Two studies (Burger 1992; Verhoest 2002) use very detailed and comprehensive sets of indicators for the measurement of management autonomy. Advantages of such elaborated indicator sets are that the measurement validity is high and that substantial information about aspects within the dimensions of autonomy is retained after processing the raw data, enabling the researcher to get a refined view on the sometimes uneven distribution of different dimensions within management autonomy (see also further in this paper). The trade-off of such a refined indicator set is the difficult problem of aggregating scores on the different indicators into one summative score, the time consuming character of such research and, consequently, the limited number of cases that can be analysed using such a indicator set.
RESEARCH
DETAILED/ COMPREHESIVE
KIND OF AUTONOMY
AGGREGATION AND SCALING
+ +++(Gathon 1991)
Formal
S (Gathon 1991: I)
+
Formal
S
+
Primarily formal/perceived Formal Perceived
S
OPERATIONALISATION
Relative efficiency studies Dunsire, Hartley en Parker 1991 ter Bogt (1998) Pollitt, Birchall en Putnam, 1998 Burger (1992),
+(+)
+++
Formal Perceived
C
I
4
Treur(1998) Verhoest 2002
+++
Formal Factual Perceived Formal or Perceived Formal
I
Neelen ++ T 1993 Künneke, + S 1991 Van Thiel +(+) Formal or perceived C 2000 Level of operationalisation: +: limited level of detailed and comprehensiveness; ++: moderate level of detail and comprehensiveness; +++: high level of detail and comprehensiveness. Aggregation and scaling: S: formal-legal status as basis for ranking; C: comparison as basis for ranking; T: typology as basis for ranking; I: weight-based construction of indexes as basis for ranking. TABLE 2: RESEARCH ON AUTONOMY AND PERFORMANCE COMPARED ON ANALYTICAL AND EMPIRICAL LEVEL
Another important aspect of comparison is that studies may differ concerning the kind of autonomy analysed.
The analysed autonomy of public organisations may be the ‘formal’ extent of autonomy as it follows out of formal regulations, laws and contract documents.
The analysed autonomy may refer to the ‘factual’ or ‘real’ autonomy from the viewpoint of the researcher. Through an ‘objective’ review of actual decisions taken on e.g. management and policy the factual decision making competencies of the agency may be delineated. ‘Factual’ autonomy may differ from formal autonomy when e.g. the agency takes decisions in matters which are formally not delegated to it or when the central governments takes decisions in matters which are delegated (Verhoest 2002).
‘Perceived’ autonomy refers to the level of autonomy of the agency as it is perceived by respondents in the agency, by respondents in the central government or by other stakeholders. It is clear that perceived autonomy may be higher or lower than the formal or factual extent of autonomy.
An important observation is that only a few studies acknowledge explicitly these important analytical distinctions (Pollitt et al. 1997; Burger 1992; Treur 1998; Verhoest 2002), although an explicit or implicit choice of the kind of autonomy studied may bias the outcome of the studies. A second observation is that quite a few of the studies stick to the analysis of formal autonomy. The studies which make a distinction between factual, formal and perceived autonomy report an important difference between the ‘formal’ and the ‘factual’ level of autonomy, or between the ‘formal’ and the ‘perceived’ level of autonomy. In the studies of Burger and Treur and Pollitt et al. it is shown that the ‘factual’ of ‘perceived’ level of autonomy in formally ‘autonomised’ agencies is lower than it could have been expected on the basis of formal-legal shifts. Verhoest (2002) reports that in the case of the Flemish Autonomous Institutions, these agencies have more ‘factual’ management autonomy concerning financial and human resources management than formally delegated to them through laws, decrees and management contracts. Moreover, the respondents of agencies and their political superiors report these differences between formal and factual autonomy when asked for their perception of formal and factual autonomy. At the methodological level the aggregation and scaling technique used is an important element (see table 2). In order to compare the extent of autonomy between several public agencies, researchers sometimes seek to rank these agencies on a range from low to high autonomy. Because autonomy may be considered to be composed of different dimensions and because each dimension may be measured by different indicators, a comparison of the level of autonomy of agencies calls for an aggregation of the outcomes on the indicators and dimensions to one overall scale of autonomy. When we look at table 7, four different techniques are to be discerned.
5
1. First, some studies (Pollitt et al. 1997; van Thiel 2000) refrain themselves from doing calculations on the raw data of their cases. Instead, these studies compare cases by tabulating the cases’ values on the different indicators and assigning a relative position of the case on an implicitly defined autonomy continuum. 2. A second technique in these studies is to equate the level of autonomy with the formal-legal status of the agencies. The formal-legal status of the analysed cases as such enables reseachers to rank the analysed cases with respect to the level of autonomy. Therefore, these studies assume an untested normative continuum of formal-legal status and the corresponding level of autonomy. 3. Third, a scale from low to high autonomy may consist of a typology combining values on different indicators or dimensions. Neelen for instance uses a typology of ‘few, moderate and many control attempts’ to sort his case observations with respect to ten indicators of management autonomy and policy autonomy. This method involves a quite rough aggregation of data considering that his initial case observations on the indicators are measured using six degrees of autonomous decision making (i.e. degrees of involvement of government in the decision making). 4. But some studies presented in table 2 (e.g. the Burger and Treur research and the Verhoest research) design more refined scales, using by scaling techniques. These techniques result in the construction of autonomy indexes, ranging mostly from 0 (no autonomy) to 1 (maximum autonomy). The selection of techniques depends of the nature of the variables involved (i.e. cardinal, ordinal or dichotomous). It is obvious that the techniques used, their level of refinement and the selection and allocation of weights, if any, have a profound impact on the ultimate order of cases compared. These latter points call for further elaboration and reference to a wider range of studies than these presented in table 3. Several researchers (already mentioned) construct autonomy indexes to compare autonomy levels between cases (van Leerdam 1999; Gilardi 2002) (Elgie 1998; Gathon 1991) (Gathon and Pestieau 1992; Pestieau and Tulkens 1990; Burger 1992; Verhoest 2002).
STUDIES (van Leerdam 1999) Gilardi (2002)
STEPS IN CALCULATING INDEXES -
Elgie (1998)
-
Burger (1992) and Treur (1998)
-
one-level aggregation assessing weighted score on item A: sum of weighted scores two-level aggregation assessing score on item A: mean per subdimension = sum of scores / number of items B: ‘overall independence’ index = sum of A / number of subdimensions two-level aggregation assessing score on item A: mean per subdimension = sum of scores / number of items B: ‘overall independence’ index = sum of (A* weights) one-level aggregation assessing degree of autonomy per item A: overall autonomy – index = sum of (score per item *
ALLOCATING AND QUANTIFYING QUALITATIVE WEIGHTS
-
-
-
-
TESTING FOR WEIGHT UNCERTAINTY
quantitative weights allocated by researcher theoretical justification for weight allocation no weights allocated by researcher no differentiated weighting, variables all have same weights
-
no sensitivity test for weight uncertainty
-
no sensitivity test for weight uncertainty
quantitative weights allocated by researcher limited theoretical justification for weight allocation on the level of dimension of autonomy, no justification on the level of subdimensions quantitative weights allocated by respondent (A) in accordance to relative importance of involved items quantifying weights done by
-
no sensitivity test for weight uncertainty
-
no sensitivity test for weight uncertainty
6
Verhoest (2002)
-
-
weights per item) / number of items three-level aggregation: assessing degree of autonomy (direct quantifying) per item A: mean per level per management field = sum of scores / number of items B: ‘autonomy-index per management field’ = A * weight per level (direct quantifying) C: ‘overall autonomy’-index = B * weight per management field
reseracher qualitative weights (per management field) allocated by expert (group of A) in accordance to relative importance of involved items quantifying weights done by researcher direct quantifying as method
-
testing weight uncertainty by quantifying the same qualitative weight set, using other methods (expected value and random)
TABLE 3: DIFFERENT WAYS TO CONSTRUCT AUTONOMY INDEXES
Table 3 lists the methodological differences among studies in constructing indices of autonomy or independence. A first observation is that the studies seem to use weighted summation as the general method to construct their indices. Literature concerning evaluation techniques and multicriteria analysis shows that technique is not neutral. Weighted summation can only be used appropriately when combining quantitative values on the variables with quantitative weights. To use weighted summation correctly the variable or indicator should be both continuous and cardinal, because the technique treats the values and weights as such. However, in the studies the variables are at most ordinal. The same can be said of the weights allocated to the variables which are also ordinal. When using qualitative values and qualitative weights as a starting point to construct indices, other techniques are more valid, such as the regime method or expected value method (Ministerie van Financiën, Afdeling Beleidsanalyse en -instrumentatie 1992b). The choice for weighted summation can be an explicit and intentional second-best choice because other techniques fail to handle many variables and weights (Verhoest 2002: appendix 1 and 2). However, most of the studies do not question the use of weighted summation, although it is important that the researcher acknowledges the possible pitfalls when using this method. One problem with weighted summation is the transition from qualitative values on ordinal variables to quantified values. In all studies reviewed, the researcher assigns quantitative values to the qualitative data, mostly by giving the different possible values a number between 0 (minimum value on the variable) and 1 (the maximum value on the variable) and by establishing an equal numeric distance between the quantified values (e.g. 0; 0.25, 0.5; 0.75; 1.00). However, it is sometimes hard to argue that each qualitative value differs to a same extent from the next possible value. Coder bias is another possible source of bias. The extent to which the researcher does the job instead of respondents of the agencies themselves, and the sources the researcher uses to judge determine the kind of autonomy that is measured (e.g. formal, factual or perceived). Another problem is the way in which relative weights are assigned to the variables and, in the case that weighted summation is used, the way in which weights are quantified. The Burger and Treur and Verhoest studies ask respondents or experts of the involved agencies to rank the variables (subdimensions or dimensions of autonomy) according to their importance for the agency’s performance. This is a way to bring reality into the constructed index. Other studies like the one of Leerdam and Elgie justify the assigned weights by referring to theory. Some give no justification, although the assigned weights have a profound impact on the overall index and the ultimate ranking of cases. Even the choice to assign no weights could be a source of bias. As to the quantifying of the weights most studies do not consider different ways or methods. In all the studies the researcher quantifies weights by assigning values that add up to 1 with the highest number referring to the variable weighted most highly. However, different techniques exist to assign a weight set within a certain ranking of weights, such as direct quantifying, expected value or random weighting (Ministerie
7
van Financiën, Afdeling Beleidsanalyse en -instrumentatie 1992a). These latter methods differ to the extent that they expect weights to be equally distibuted. The choice of one of these methods should be deliberate. Linked to the latter point, multi-criteria analysis literature point at the importance of testing indexes and ultimate ranking of cases for weight sensitivity. Such testing could be done at two levels. First, one could test if the ultimate ranking of cases changes when one changes the qualitative weights assigned to the underlying variables (by calculating the consequences of such change or by calculating uncertainty percentages). On a second level, sensitivity tests could be done to test the impact of the way the weights are quantified. Verhoest performed tests on this second level by recalculating indexes and rankings with quantified weights based on the expected value method and the random method and by comparing these indexes and rankings with those he has calculated using weights quantified by the ‘direct quantifying’ method. A last difference in index construction is the number of levels of aggregation. The studies range from one-level aggregation to three-level aggregation. The number of levels is clearly a trade-off. Fewer levels of aggregation reduce the number of possible biases which follow from choice concerning values and weights and from calculations. More levels of aggregation permit insights into changes of autonomy on the different subdimensions and dimensions of autonomy. The information content of such multi-level indexes increases with each level, as does the possibility of bias. Such biases, however, can be overcome by explicitly dealing with them. In this section we have shown that contemporary studies on autonomy and performance vary considerably with respect to the way they handle autonomy on an analytical and methodological way. We can argue that the analytical and methodological choices are not neutral with respect to the outcomes of such studies. However, these choices are not always well-documented in the studies.
4. Kind of complex indexes used in the COBRA research The indexes used in the Leuven part of the COBRA research are to be distinguished in (for examples see table below): (1) Re-coded variables, a necessary first step to make indexes. (2) "Logical indexes", built following a certain inherent logic. No consideration is made about statistical issues: the index is constructed (variables are merged in a new variable) because this seems logical. (3) Indexes based on statistical significance (correlation and alpha); these indexes are computed by calculations on raw data or recoded variables, and result is an valid scale (correlation and alpha).: • Based on unweighted summation with quantitative values assigned to scale by researcher: • Based on unweighted summation with quantitative values assigned by respondent In fact these are “theoretical-deductive indexes, because it is in advance decided what variables should be in the index. Afterwards, the index is validated by using statistics (correlation and alpha). (4) Index (aggregating several variables) based on factor analyses These are “empirical-inductive indexes”: a selection of variables (e.g. concerning autonomy) is taken, and then factor analyses are performed. This results in some empirical factors, our indexes, on which every case has its score. (5) complex index (aggregating several variables) based on cluster analysis This is “empirical typology construction”: cases are clustered along two or more variables. Take for example two variables on autonomy: a possible outcome is then e.g. four clusters of cases (typologies): high-high, high-low, low-high, low-low.
8
5. Complex indexes of autonomy within the COBRA research Scope The concept of autonomy is very diverse, as the literature and recent research shows. Without wanting to go into detail (this is not a conceptual note) we see diversity on at least three levels: - Content: • management autonomy; e.g. HRM or financial management • policy autonomy • other kinds of autonomy (e.g. financial, structural etc.) -
-
• • • •
Qualification Formal autonomy, as defined in laws, decrees, contracts and other formal documents Factual or ‘real’, as observed by the researcher in reality Perceived: as the actors perceive (estimation of one own’s autonomy) Desired: as desired by the granting (principal) or receiving (agent) actor
• • • •
Relations agent (in itself a heterogenous concept) vis-à-vis political principals agent vis-à-vis administrative principals agent vis-à-vis supranational governmental actors agent vis-à-vis “third” actors such as stakeholders etc.
Measurement In the Flemish COBRA survey, we have assessed management autonomy and policy autonomy vis-àvis political and administrative oversight authorities. As to “alternative” autonomy, also structural and financial steering were investigated for example. The survey resulted in raw data about organizational autonomy (amongst other concepts and variables investigated) of about 80 public organizations, ranging from core departmental organizations to autonomous agencies. In order to deal with these data (constructing variables for research purposes), a substantial number of data were recoded or aggregated in indexes.
Construction and Validation Indexes are checked for validity in two broad manners; empirical and theoretical. Empirical validations are about alpha’s (checking internal reliability of the scale) or correlation analysis (between items & between items and index). More sophisticated techniques that are bottom-up are (1) factor analysis of raw data (resulting in empirical factors that contain variables that “measure the same construct”) or (2) cluster analysis. Cluster analysis groups organizations (respondents) according to their position on two ore more variables, which leads to an empirical typology of organizations on an aggregated level. Theoretical validations is about relying on existing knowledge (from theory, or literature) to construct a “theoretically logic” index. In the table below, some examples of autonomy indexes from the Flemish COBRA survey are listed, as well as the characteristics of the indexes.
9
Examples of autonomy indexes in survey research INDEX Recoded variables Smaloan Smatar Smapar Omaloan Omatar Omapar Patamr Patadr Painmr Paindr Goaldoc Goal Indic Meas Eval SB TOTBOA PERCGIB
DESCRIPTION
VARIABLES
TYPE
VALIDATION
Strategic management autonomy for taking loans
Recoded variable from raw data
RV
Strategic management autonomy for setting tariffs Strategic management autonomy for participation in other persons Operational management autonomy for taking loans Operational management autonomy for setting tariffs Operational management autonomy for participation in other persons Policy autonomy choosing target groups - minister Policy autonomy choosing target groups - department Policy autonomy choosing instruments - minister Policy autonomy choosing instruments - department Document in which goals are specified Institution that sets goals Indicators for measuring goal attainment Institution that measures results Institution that evaluates results Existence of sanctions and rewards Total number of board members % of board members that are governmental representatives
Recoded variable from raw data Recoded variable from raw data
RV RV
Logical exercise: preparing raw data for further analysis Idem Idem
Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data
RV RV RV
Idem Idem Idem
Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data Recoded variable from raw data
RV RV RV RV RV RV RV RV RV RV RV RV
Idem Idem Idem Idem Idem Idem Idem Idem Idem Idem Idem Idem
Strategic HRM autonomy
4 survey items
TD
Operational HRM autonomy Strategic FM autonomy Operational FM autonomy Policy autonomy vis-à-vis department Policy autonomy vis-à-vis minister Index of result steering
4 survey items Smaloan, smatar, smapar Omaloan, omatar, omapar Patadr, paindr Patamr, painmr Goaldoc, Goal, Indic, Meas, Eval,
TD TD TD TD TD LI
Correlations of sub-variables, correlations of sub-variables with index, alpha test for internal reliability Idem Idem Idem Idem Idem Logical index, cumulative logic in the result
Level 1 index SPA OPA SFA OFA PAD PAM Resstc
10
Indexa Govinboa
Index of ex ante autonomy Percentage of governmental representatives in board
Strumana
Index of steering organisation by means of appointing management Index of steering organisation by means of evaluation management Index of financial steering
Evalmana Finstc
Sb 2 survey items Totboa, percgib
LI LI
steering cycle Logical index (detailed or general steering) Logical index; score between 0 and 1, depending on % Logical index; score between 0 and 1, depending on who appoints Logical index; score between 0 and 1, depending on who evaluates Logical index; score between 0 and 1, depending on % of budgetary means that come from government
Survey items
LI
Survey items
LI
Survey items
LI
OPA, OFA SPA, SFA PAD, PAM Resstc, Exa1, Exa2 Govinboa, strumana, evalmana, finstc
EI EI EI EI EI
Cluster analysis: empirical validation Cluster analysis: empirical validation Cluster analysis: empirical validation Cluster analysis: empirical validation Cluster analysis: empirical validation
EI
Factor analysis: empirical validation
EI EI EI
Factor analysis: empirical validation Factor analysis: empirical validation Factor analysis: empirical validation
Level 2 index Typologies OMA SMA PA Common steer Altern. steer
Clusters of operational management autonomy Clusters of strategic management autonomy Clusters of policy autonomy Clusters of “common” steering (ex ante & results) Clusters of “alternative” steering (structural & financial)
Factors SPA OPA FMA POLA
Factor found after factor analysis of 12 survey variables and 3 variables → Idem Idem Idem
Table 1 *RV: Recoded variable *TD: theoretical-deductive index; ex ante composed, based on theoretical insights (we assume e.g. that variables of HRM autonomy can be integrated in an aggregated HRM autonomy index). *LI: logical index *EI: empirical-inductive index; statistical manipulation of variables via factor analysis or cluster analysis. E.g. putting all autonomy variables in a factor analysis, and see what factors we observe. For cluster analysis, we cluster organizations along some autonomy variables (e.g. in the simplest way, for operational management autonomy we can cluster the observations (organizations) in four empirical typologies; both OPA and OFA high, both OPA and OFA low, OPA high – OFA low, OFA high – OPA low)
11
Methodological strengths and weaknesses of our autonomy indexes Confronted with the problems of index construction and validation we have developed some ways to tackle the different issues. However, some problems remain unsolved and need further consideration. In particular we focus here on our indexes of autonomy, both managerial and policy autonomy.
1. Concept and selection of variables Management autonomy The concepts of both financial and personnel management autonomy are operationalised in a moderate detailed manner with 4 variables for personnel management autonomy and 5 for financial management autonomy. These operationalisations are selected carefully based on extensive case study research which used much more detailed and extensive sets of variables to measure both concepts (Verhoest 2002). When checked for alpha and factor analysis the operationalisations of both concepts seems to be valid. The extensive case study research (Verhoest 2002) enabled us to construct measures for managerial autonomy which were not to extensive and detailed, while covering a relevant selection of dimensions of the concept. The extensive case study research (Verhoest 2002) allowed us also to distinguish between formal, factual and perceived levels of autonomy. The survey contains perceived levels of autonomy, which seems to refer to factual levels of autonomy, rather than formal. Policy autonomy The operationalisation of policy autonomy in two variables is limited in scope, based on theoretical notions. For this concept we could not rely on previous extensive research to develop measures for policy autonomy. However, our alpha and factor analysis on these variables and the resulting concept show that the measures are robust. Moreover, in his ongoing Ph. D. Bram Verschuere is studying the measurement of policy autonomy thoroughly, using case study methodology and distinguishing between formal, factual, perceived and desired levels of policy autonomy. Again, case studies will be used here to complement and validate survey research and to develop robust measurements of autonomy.
2. Quantifying values In our research we have indexes based on variables with quantitative values assignes by the respondent (see e.g. dimensions of culture and performance). For our indexes on management and policy autonomy, we use variables with quantitative values assigned to scale by the researcher. Certainly for the variables on policy autonomy2 this is obvious: 1 the organisation make most of the decisions itself, parent ministry only slightly involved in the decision making process and sets few restrictions. .75 The organisation make most of the decisions itself, after having consulted parent ministry .50The organisation makes most of the decisions itself in correspondence with conditions or restrictions set by parent ministry .25 Parent ministry makes most of the decisions, after having consulted the organisation
.00 -
Parent ministry makes most of the decisions, independently of the organisation.
There are two potential flaws: 1) the scale is not complete as the option: “the organisation make most of the decisions itself, parent ministry not slightly involved in the decision making process and sets no restrictions.” Is missing. 2) Moreover, we cannot say for sure that the distance between all the options is equal, although by 2
With respect to personnel and financial management autonomy we mostly use dichtomous variables (operational autonomy for appointing employees: yes or no). 12
assigning the quantitative values we treat them as such (e.g. consider the second and third option in the scale). To reduce this problem we have developed a new scale for the future surveys which is more logic and which implies more equal distances between the different options: Organization takes most of the decisions itself, minister/parent department is not involved in the decision making process and sets no restrictions Organization takes most of the decisions itself, minister/parent department is only slightly involved in the decision making process and sets only minor restrictions Organization takes most of the decisions itself, after having explicitly consulted the minister/parent department or under explicit conditions or restrictions set by the minister/parent department The minister/parent department takes most of the decisions, after having consulted the organization The minister/parent department takes most of the decisions, independently of the organization Nor the minister/parent department, nor the organization decides on this matter, since the involved legislation leaves no room for discretion on that matter
However, the fundamental problem remains: By assigning quantified values options are treated as having the same distance, while in reality respondents may not perceive the differences between the options as such. Moreover, by assigning quantified values, variables which are ordinal, may be perceived as being cardinal. Therefore, we avoid to apply statistical methods for cardinal values on our variables and indexes of autonomy, but we try to stick to methods which are valid for ordinal variables (e.g. nonparametric methods like kruksal-wallis).
3. allocation of weights to variables Usually our indexes are composed of unweighted variables, indicating that the used variables have an equal importance. For example, the index of strategic personnel management autonomy is composed of variables with respect to the extent to which organizations can set general rules on: • Level of salaries • General criteria for promotion • Way of evaluating personnel
• Way of appointing personnel • ( Criteria for dismissal) Based on alpha’s and factor analysis it is clear that the four variables measure the same concept. However, by assigning no weights we assume that these four issues of personnel management all have equal relevance and importance for a strategic personnel policy in an organization. However, one could hypothesize that for some organizations certains issues like salary and appointment will be more important for their autonomy. Moreover, in the comparison between personnel management autonomy between agencies in Flanders and Ireland, it appeared to be that decisions concerning personnel evaluations in Ireland were less relevant than in Flanders. In Flanders, personnel policies of public organizations link up salary, promotion, evaluation and appointment in a more coherent way than in Ireland. One could make the same reasoning with respect to the relative weight assigned to personnel management autonomy and financial management autonomy, when discussing the management autonomy of organizations (e.g. compare an organization with labour-intensive tasks versus an organization with tasks based on huge capital investments). And also for policy autonomy, decisions on target groups may be perceived by organizations as being relatively more important than on policy instruments, or vice versa. Until now we have assigned no weights to the different variables because we did not have any data about the preferences of organizations. However, one possible way to tackle this problem would be to ask in the survey which is the perceived ranking of issues (salary, promotion, evaluation, appointment) with respect to their importance and relevance for the autonomy of the organization involved. Base on that information one could calculate an index of personnel autonomy for each organization which takes into account the relative weights that the respective organizations assign to the different variables (see e.g. Burger 1992). One other way would be that one respondent per country assigns a ranking of the issues for the organizations surveyed in that country. For example such a method has been used in earlier case study research (Verhoest 2002), where a representative (I;e. head of the interest group of autonomous agencies) was asked to rank the different issues of autonomy on their relative importance for Flemish agencies. With such a data we could compute country-specific indexes
13
taking into account differences between countries with respect to the relative importance of autonomy issues for the agencies in these countries.
4. Techniques for computing indexes In our research we try to use different methods to construct indexes of the same concepts. We then check if they validate one another. A clear example is to compose autonomy indexes by unweighted summation and and then checking them by factor analysis. But for some indexes we should think of ways to develop this further (e.g. ex post control).
5. Checking for sensitivity By combining different method for index calculation we can study the sensitivity of our indexes with respect to the method. But due to weaknesses discussed above with respect to allocation of weights and quantified values, checks for sensitivity with respect to these aspects are underdeveloped in our research at this moment.
6. Levels of aggregation Because of the risk for multiplying bias problems, we refrain to compute indexes which contain several levels (i.e. an index based on a set of indexes). If we do compute level 2 indexes we use cluster or factor analysis, rather than unweighted summation, because cluster and factor analysis do respect the original data with respect to their quality. For example, we do not merge the indexes on operational HR management autonomy and strategic HR management autonomy in one index of HR management autonomy. Likewise, we do not try to establish an overall management autonomy index of organizations, based on indexes for HR and financial management autonomy. Trying to calculate such complex indexes we would increase the problems of assigning weights, quantified values, changing the ordinal value of our variables enormously, which would jeopardize the quality of our research results to a high extent.
6. Conclusions and next steps From what is described above it is clear that the use and calculation of indexes should be considered as a set of decisions and important trade-offs. In our COBRA research we tried to take into account some potential flaws of indexes and to avoid them. But at the same time we restrict ourselves with respect to possible statistical techniques that we can use and the kind of indexes that we can calculate. Indexes clearly have limitations but also opportunities (see below). The crucial questions is how to decrease the limitations and how to increase the opportunities of the indexes that we use. By moving to the next stage of COBRA research with comparative research across countries using similar indexes, new opportunities arise to strengthen the validity of our indexes and concepts further. For example, we could compare the validity of the autonomy indexes in different countries (e.g. by comparing alphas and by doing factor analysis. Moreover, it should be clear from this text that index calculation is not a neutral exercise, but a procedure with several possible biases. In our research we need to be clear about the choices we make and the steps we take to calculate and validate indexes. Overall, this is an aspect in public management research on autonomy which is frequently neglected (also by ourselves).
Limitations For doing statistical analysis -
type of variable (nominal, ordinal, ratio) limits statistical opportunities
14
-
use of statistical techniques with indexes (e.g. regression) if one item misses for one observation, index cannot be constructed: decreasing N …
Empirically -
Loss of data after aggregation Survey: perceptions or facts? Aggregating variables of different types needs a recoding first Quantification of qualitative data Weighted versus unweighted + who weights? …
Opportunities -
empirical validation of theoretical constructs (e.g. is there difference between types of autonomy, as conceptually and theoretically presumed) combining indexation methods for increased validity of the index aggregating data in indexes gives clearer picture, without loosing the underlying meaning of the index (underlying variables) aggregating data can lift variable to a higher analytical status (from nominal to ordinal, rational order via factor score) …
15
Reference List Beuselinck Eva and Notebaert Karolien (2003) . Constructie van synthetische performantieindicatoren. Leuven: unpublished manuscript. Bogt ter, Hendrik J. 1998. "Neo-Institutionele Economie, Management Control En Verzelfstandiging Van Overheidsorgansiaties: Overwegingen Voor Verzelfstandigingen Effecten Op Efficiëntie En Financieel-Economische Sturing." University of Groningen, Groningen, the Netherlands. Borcherding, Thomas E., Werner W. Pommerehne, and Friedrich Schneider. 1982. "Comparing the Efficiency of Private and Public Production: the Evidence From Five Countries." Zeitschrift Für Nationalökonomie - Journal of Economics(supplementum 2):127-56. Borger De, B. and K. Kerstens. 1994. "Produktiviteit En Efficiëntie in De Belgische Publieke Secotr: Situering En Resultaten Van Recent Onderzoek." Studiecentrum Voor Economisch En Sociaal Onderzoek, Vakgroep Publieke Economie. Rapport 94/304. 38 . Burger, Yvonne D. 1992. Tussen Realiteit En Retoriek: Decentralisatie En Autonomisering in De Praktijk. Delft: Eburon. Burger, Yvonne D. and J. H. F. Treur. 1996. "Decentralisation and Autonomisation As a Strategy for Increased Effectiveness in the Public and the Private Sector." Society and Economy in Central and Eastern Europe 96(2):114-27. Fölster, Stefan. 1997. "Ist Der Systemwechsel in Schweden in Gefahr? Erfahrungen Mit Privatisierung, Deregulierung Und Dezentralisierung." Pp. 125-34 in Reformen Des Öffentlichen Sektors in Skandinavien. Eine Bestandsaufnahme, eds. Claudius H. Riegler and Frieder Naschold. BadenBaden: Nomos Verlagsgesellschaft. Gathon Henry-Jean. 1991. La performance des chemins de fer européens: gestion et autonomie. Thèse de doctorat. University de Liège. Gathon, Henry-Jean and Pierre Pestieau. 1992. "Faut-Il Encore Mesurer La Performance Des Entreprises Publiques?" Annals of Public and Cooperative Economics ?(?):621-44. Gilardi, Fabrizio. 2002. "Policy Credibility and Delegation to Independent Regulatory Agencies: a Comparative Empirical Analysis." Journal of European Public Policy 9(6):873-93. Künneke, Rolf W. 1991. "Op Armlengte Van De Overheid: Een Theoretisch En Empirisch Onderzoek Naar De Effecten Van Verzelfstandiging Op De Efficiëntie Van Openbare Nutsbedrijven." dissertatie, Universiteit Twente, Vakgroep Bestuurskunde, Enschede. Millward, Robert and David M. Parker. 1983. "Public and Private Enterprise: Comparative Behaviour and Relative Efficiency." Pp. 199-274 in Public Sector Economics., editors Robert Millward, David M. Parker, Leslie Rosenthal, Michael T. Sumer, and Neville Topham. London and New York: Longnam. Ministerie van Financiën, Afdeling Beleidsanalyse en -instrumentatie. 1992a. Evaluatiemethoden, Een Introductie. Vierde herziene druk ed. Den Haag: SDu Uitgeverij. ———. 1992b. Evaluatiemethoden, Een Introductie. Vierde herziene druk ed. Den Haag: SDu Uitgeverij. Neelen, Geert H. J. M. 1993. "Principal-Agent Relations in Non-Profit Organisations: A Comparative Analysis of Housing Associations and Municipal Housing Companies in the Netherlands." Universiteit Twente, Faculteit der Bestuurskunde, Entschede. Pestieau, Pierre and Henry Tulkens. 1990. "Assessing the Performance of Public Sector Activities: Some Recent Evidence From the Productive Efficiency Viewpoint." CORE Paper. 58 p. Pollitt, Christopher, Johnston Birchall, and Keith Putnam. 1997. Opting Out & the Experience of Self-
16
Management in Education, Housing and Health Care. ESRC Local Governance Programme Working Paper 2 ed. Glasgow: The Local Governance Programme of the Economic & Social Research Council. Pollitt, Christopher, Johnston Birchall and Keith Putnam. 1998. Decentralising public service management. Hampshire: Macmillan, 195 p. Treur, J. H. F. 1998. "Centralisatie En Decentralisatie Bij De Nederlandse Politie: Over Contractuele Aansturing En Resultaatverantwoordelijkheidsstelling Bij De Nederlandse Politie." dissertatie, Rijksuniversiteit Groningen, Koninklijke Vermande, Lelystad. van Leerdam, J. 1999. "Verzelfstandiging En Politieke Economie. Over De Betekenis Van Het NieuwInstitutionalisme Voor De Instellingen Aansturing Van Zelfstandige Bestuursorganen." dissertatie, Katholieke Universiteit Brabant, Delft. van Thiel, Sandra. 2000. "Quangocratization: Trends, Causes and Consequences." Interuniversity Center for Social Science Theory and Methodology (ICS), Utrecht. Verhoest Koen 2002. "Resultaatsgericht Verzelfstandigen: Een Analyse Vanuit Een Verruimd Principaal-Agent Perspectief. " Catholic University of Leuven, Faculty of Social Sciences, Leuven.
17
