Students' perceptions on the influence of institutional

This article was downloaded by: [Leite, Denise Balarine Cavalheiro] On: 21 July 2009 Access details: Access Details: [subscription number 913217794] Publisher Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Assessment & Evaluation in Higher Education Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t713402663

Students' perceptions on the influence of institutional evaluation on universities Denise Leite a; Rui A. Santiago b; Cláudia S. Sarrico b; Cecília Loréa Leite c; Marlis Polidori a a Federal University of the Rio Grande do Sul, Brazil b University of Aveiro, Portugal c Federal University of Pelotas, Brazil Online Publication Date: 01 December 2006

To cite this Article Leite, Denise, Santiago, Rui A., Sarrico, Cláudia S., Leite, Cecília Loréa and Polidori, Marlis(2006)'Students'

perceptions on the influence of institutional evaluation on universities',Assessment & Evaluation in Higher Education,31:6,625 — 638 To link to this Article: DOI: 10.1080/02602930600760264 URL: http://dx.doi.org/10.1080/02602930600760264

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

Assessment & Evaluation in Higher Education Vol. 31, No. 6, December 2006, pp. 625–638

Downloaded By: [Leite, Denise Balarine Cavalheiro] At: 19:59 21 July 2009

Students’ perceptions on the influence of institutional evaluation on universities Denise Leitea, Rui A. Santiagob*, Cláudia S. Sarricob, Cecília Loréa Leitec and Marlis Polidoria aFederal cFederal

University of the Rio Grande do Sul, Brazil; University of Pelotas, Brazil

bUniversity

of Aveiro, Portugal;

Assessment 10.1080/02602930600760264 CAEH_A_175985.sgm 0260-2938 Original Taylor 602006 31 [email protected] RuiSantiago 00000December and & Article Francis (print)/1469-297X Francis & Evaluation Ltd 2006 Ltd in Higher (online) Education

There are many studies about the experiences of higher education students, but few analyse their representations of the governance and the management of their institutions. Our study will describe, analyse and compare students’ representations of institutional evaluation at three institutions in Portugal and Brazil. Our results, based on an open questionnaire which sought to collect students’ views on institutional evaluation, show that students’ views are neither homogeneous nor do they simply reproduce the dominant external perspectives on the subject. A significant amount of students thought that evaluation is a legitimate exercise, resulting from a political decision, which is useful to improve the quality of universities and the relationship between students and academic staff. Some students also recognize that evaluation will lead to institutional comparisons, functioning as a mechanism of control, regulation, monitoring, and possibly of standardization. It is also noted to be ‘an impossibility’, since it can never comprehend the existing diversity of institutional performance.

Introduction Under the converging effects of financial restrictions, rising expectations and social demand, growing influence of the new economy and the erosion of its symbolic capital, higher education has been exposed to the influence of strong outside pressures for change. Universities are compelled to assume greater social responsibilities for the value of their teaching and research activities and promote their adaptability within the economy and the market (Meek, 2002, 2003; Reed, 2002; de Boer, 2003; Massen, 2003; Salminen, 2003). At the same time, to a certain extent, these *Corresponding author. Social, Juridical and Political Sciences, 3810-193 Aveiro, Portugal. Email: [email protected] ISSN 0260-2938 (print)/ISSN 1469-297X (online)/06/060625–14 © 2006 Taylor & Francis DOI: 10.1080/02602930600760264


626 D. Leite et al. constraints have given rise to profound adjustments of the perceptions of higher education’s ultimate goals. As in most western countries, both in Portugal and Brazil the meaning of these ultimate goals seems to be moving away from the approach based on higher education’s social, cultural, critical and axiological relevance (Barnett, 1990, 2000; Miller, 1995; Meek, 2002; Reed, 2002; Leite, 2002; Santiago and Carvalho, 2004), towards another approach more related to the economy. The translation of this new political, economic and educational environment into practices of internal regulation, based upon quality evaluation, has increasingly lead universities to query students about their satisfaction, attitudes, needs and experiences at various academic levels. The predominant perception, namely at the governmental and university top-management level, is that the evaluation by the students of the curricula, teaching and academic experiences constitutes an important element in the development of quality processes. Not taking into consideration the feedback of the students, and not translating it into correct measures, are factors that emerge as important failures in these processes (Leckey & Neiel, 2001). Therefore, it is expected that students express their insights on the quality of the offered services (Harvey & Green, 1993) or that, as consumers (Green, 1994), clients or users they should be involved in the dialogue regarding the quality of their educational activities. This study chooses as a research theme the students’ own perceptions of the evaluation of universities. In specific terms, it is focused on the analysis of political, institutional and educational implications that Brazilian and Portuguese students assign to the institutional evaluation and its objectives. It is expected that the results of this analysis will ultimately allow a better understanding of students’ predisposition to involve themselves in the institutional evaluation processes of higher education institutions. First, we present a detailed explanation for our theoretical options, namely the choice of research theme and our preference for the concept of actor to characterise the institutional position of the student, rather than that of client, user or consumer. Second, we discuss the research methodology used in the study, of an essentially qualitative nature, drawing on the different peculiarities of the universities’ evaluation systems in Portugal and Brazil. These peculiarities might help us understand why the Brazilian students’ discourse often has more political and social tonalities. However, in essence, the majority of the preoccupations with the evaluation of universities are common to the students of both countries. Finally, we present and discuss the results of a content analysis of the students’ answers to an open questionnaire, which was administered in the three universities involved in the project— University of Aveiro (Portugal), Federal University of Rio Grande do Sul and Federal University of Pelotas (Brazil). Finally, we discuss the conclusions derived from the analysis. Students as key actors in the institutional evaluation process The great majority of studies have centred on the analysis of students’ experiences. These studies and not only those under the influence of managerial pretexts, have


The influence of institutional evaluation on universities 627 focussed on the following domains: teaching and teacher’s evaluation (Sander et al., 2000), teaching assessment (Santhaman & Hicks, 2002) and teaching and research evaluation (Jenkins et al., 1998); students’ perceptions on their education, studies guidance and academic outcomes (Entwistle & Ramsden, 1983; Ramsden, 1985; Biggs, 1987; Leite, 1990, 1992; Tavares et al., 1998; Santiago et al., 2001; Lizzio et al., 2002), academic systems (Bos & Tarnai, 1999) and the academic tasks of higher education teachers (Rousseau et al., 1995). The students’ satisfaction concerning the features of their educational experience (Harvey & Green, 1993) has also been, without a doubt, a relevant field of reflection on the organizational, educational and curricular transformations. Some authors recognize the need to involve the students in participative evaluation processes of the universities’ activities (Powell et al., 1997; Reynolds & Trehan, 2000) or through dialogue, to induce the students to participate in the democratisation of their education’s quality (Green, 1994). In spite of the intensive research investment described above, the students’ perceptions on the universities’ evaluation have not been elected as a study-object. This may be due to the fact that students are not traditionally seen as actors with the ability to evaluate the evaluation, and their opinions might have been disregarded simply because of their interference in the professional expertise of academics. In fact, in the great majority of the evaluation processes, students are mostly considered just as a source of external information. This denotes an important limitation of the concept of evaluation guided towards change and involving the institution as a whole. However, it is possible to imagine that students are capable of reflecting upon their institutional experiences and of questioning their universities’ evaluation pretexts and processes. As we know, the analysis of the different issues of institutional evaluation does not end in the field structured by the impositions of the evaluator state (Neave & Van Vught, 1994) or by market devices. However, even if these impositions are not there, the evaluation of universities, regardless of its model, has mostly been an external view that can never be included in the institutions’ ethos. This view is always based upon the formal patterns (which are considered ‘technically neutral’) or upon a wider normative framework, which in most cases produce phenomena of evaluative socio-centrism that naturally distort the assessment of the characteristics and the institutional activities. This does not mean that the evaluators are not concerned with helping the higher education institutions face their difficulties. However, the perception of the evaluation, limited to its characterization as a qualified organizer—defined as only the transformation of valid data into useful information in a methodical and value-added way (Morosini & Leite, 1997)—is not truly useful if its processes and results are not appropriated by the actors and widely integrated in the higher education institutions. This presupposes a profound reflection about the social roles of evaluation (evaluation of the ‘why’ and ‘what for’) and about the place it takes in the system and institutions, mainly with concern to its meaning as regards political decision making and to the institutions’ behaviour and academic actors.

628 D. Leite et al.


The national contents of higher education in Brazil and Portugal The evaluation of universities occurs as a part of Brazilian and Portuguese national systems of evaluation. In Brazil, the evaluation of postgraduate studies and research, both in the public and private sectors, is undertaken, since the 1970s, by CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), an agency for the training of higher education academic staff, and by CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico), the National Counsel of Technological and Scientific Development, respectively, by peer review and supported with a set of recognised performance indicators. These evaluations are perceived by the academic community as a quality assurance process, since the criteria are largely consensual, clearly known, and the results regularly publicised. Graduate teaching is also the object of periodical evaluations by the Ministry of Education. It can involve a process of auto-evaluation, according to criteria defined by a plan for the institutional evaluation of Brazilian universities (PAIUB - Plano de Avaliação Institucional das Universidades Brasileiras). This plan, suggested by the rectors of the Brazilian public universities, was being used, at some point, by 138 of the 153 public universities, but has been losing ground over the years. This model, essentially based on the processes, is similar to the Portuguese evaluation model of graduate degrees, which is presented later. Since 1995, however, the evaluation of graduate teaching has been done by a national exam of degrees (ENC - Exame Nacional de Cursos). This exam is administered to students at the end of their degree, both in the public and private sectors. This so called ‘Big Final’ (‘Provão’), as it is popularly known, constitutes an external form of evaluation, principally centred on the results of students. In each degree submitted to this exam, the teaching conditions are also examined through a visit by external experts of the scientific area of the degree. The results of these degree evaluations form the basis of a national ranking, which is widely publicised in the country. In addition to these evaluations, the evaluation system of the federal universities also includes the evaluation of academics through a merit pay system (GED - Gratificações de Estímulos à Docência), which may add value to the annual salary of academics. All of these evaluations have supported several pedagogical and institutional development projects. It is worth noting that the different evaluative processes, which have been going on uninterrupted for a considerable number of years, have favoured the communication between degrees and units and have given rise to managerial forms, which give more emphasis to the notions of quality, efficiency and competition. In Portugal, a law was passed in 1994 that established the basis for a system of evaluation and monitoring of higher education institutions. The adopted evaluation system was largely inspired by the Dutch model. It focuses on the scientific, pedagogic and cultural performance of institutions, and it integrates a process of autoevaluation followed by a process of external evaluation. The evaluation process was the object of an agreement between the ministry responsible for higher education and the Council of Rectors of Portuguese Universities (CRUP - Conselho de Reitores das Universidades Portuguesas) through its Portuguese Universities’ Foundation (FUP Fundação das Universidades Portuguesas). This last institution is recognised as the


The influence of institutional evaluation on universities 629 representative entity of public universities in the development of the evaluation processes. The private universities and the public and private polytechnics have their own evaluation systems, similar to the system first developed for public universities. After the agreement had been signed, the Portuguese Universities’ Foundation approved a Council for National Evaluation, supported by several consultative commissions, which monitors the development of evaluation in public universities. The Portuguese evaluation of degrees is only carried out for graduate degrees and it can be characterised by a developmental or ‘formative’ evaluation, more centred on the processes and on institutional ‘consultation’ than on results (Santiago et al., 2003). Recently, other laws have been passed on the development and quality of higher education and its public financing, which value more the results and the ‘summative’ dimension of the evaluation of universities. As to the evaluation of research, there is a separate system with procedures similar to the evaluation of graduate degrees—auto-evaluation followed by external evaluation by teams of national and foreign peers—which covers the scientific production of research units, centres and laboratories recognised by the ministry responsible for higher education. The results of this evaluation produce immediate consequences in the funding allocations to the evaluated units. The funds distributed are proportional to the position that the units can attain in a classification scale of five points (from poor to excellent). Research methodology The results presented in this article showing the analysis of students’ perception of the universities’ evaluation have been extracted from a wider project, which stresses the relationship between evaluation, organizational learning and management of universities. The collected data had two main features: (a) the insight of the actors responsible for the universities’ top management—rectors and vicerectors of the three universities that participated in this project (Federal Universities of Rio Grande and of Pelotas, in Brazil, and University of Aveiro, in Portugal); (b) the students’ insight about evaluation, containing an intentional sample, defined according to the number of courses existent in each of the universities. The interviews carried out with the top academic managers of each university will not be the subject of analysis in this article. The questionnaire administered to the students had three open questions, the last one being subdivided in three possible answers (see Table 1). The first question sought to solicit an opinion on the evaluation of universities, without willingly identifying its format (internal or external evaluation, educational evaluation, and others). The second question induced the connection of evaluation to the likely improvements observed in the degrees. The last question wondered if the evaluation results originated improvements and, if so, of what sort: in teaching, research, or co-operation with society. As to the sample used, in the case of the Federal University of Rio Grande do Sul (Brazil), the questionnaire was sent to all students via its intranet (as in the

630 D. Leite et al. Table 1.

Questions in the questionnaire

What is your opinion regarding the evaluation of universities? Do you think that evaluation of the university produces or will produce improvements in your degree? 3. What results of the evaluation of the university will be a driver for change? Where? To what extent? In teaching: In research: In university-society co-operation: Other:


1. 2.

other two universities, all students have an institutional e-mail address). From the answers thus obtained a random sample would be extracted. However, the number of filled in questionnaires was so low (67 in about 25,000 students), that it prompted the researchers to change the data collection strategy for the other two universities; where the questionnaire was administered directly to a random sample of students. The details of the study samples for the three universities involved are shown in Table 2. The answers obtained from this questionnaire were entered into the QSR NUD*IST software and were systematically analysed in several meetings by the researchers in Brazil and in Portugal. The researchers have met to consider the categorisation of the content analysis of the answers. These were individually examined by each researcher, who then discussed together its categorisation until a consensus was reached. Thus the themes extracted from the discourse of the students were a result of a consensus building exercise among the researchers. These items were grouped according to their key-ideas. From this process, different themes originated, described by the students’ words, which were then classified in categories, inserted in the software database, and structured in coded knots. As underlined before, there were not many differences, apart from the above mentioned political and social emphasis, between Portuguese and Brazilian students. For this reason, the results of both groups are presented together. Additionally, there were no significant differences in the results in relation to the degree, year of degree, gender or age of the students. Table 2. Institution Universidade Federal Rio Grande do Sul Universidade Federal de Pelotas Universidade de Aveiro Total

Study Samples Number of students

Total of degrees involved

67 269 130 466

28 21 6 55

The influence of institutional evaluation on universities 631 Findings In this article we only analyse the meaning attributed by the students to the concept of institutional evaluation (first question of the questionnaire) and its impact on degree improvement (second question of the questionnaire). The main themes, taken out from the content analysis are as follows: evaluation and teachers, constructive evaluation, universities’ control and regulation by the state, evaluation validation and legitimacy, accountability, discrepancy and comparison between universities, and segmentation/fragmentation of assessment.


Evaluation and teachers Generally, some students consider that the relationship they have with their teachers, in the context of the classroom, influences their perspective of the university, namely, of the evaluation of the university. The students believe the evaluation process should contemplate a higher proximity to their opinions and expectations, which would favour closeness with the institution. The assessment questionnaires of degrees are ‘cold’ and impersonal, and do not take into account students’ subjectivity or the degrees’ relational, social and cultural dimensions. In relation to this, a student says: I believe the questionnaires we answer at the end of each course help in this evaluation, but they are inconclusive. They do not include the subjectivities of relationships in a classroom, which ultimately influence the results … (History student)

The majority of these students consider, however, that universities’ evaluation is liable to induce enhancements in the teachers’ performance, namely because: the results are a feedback to the teachers and they are more involved in the academic success of the students’; it induces an increase in the teachers’ effort; universities are more concerned about teaching and teachers’ recruitment. But some students show deep doubts about the positive effects of universities’ evaluation concerning the performance of teachers. Evaluation would only originate results if it were centred in the teaching methods or in the teachers’ quality: In my opinion, evaluation should also include teachers, since they also need monitoring of their teaching methods and knowledge, especially in practice rather than in theory. (Education student) To tell you the truth, we could count on the fingers of my hand the teachers who are really teachers; most of them have the knowledge, but don’t know how to teach or have the least of didactic training. (Mechanical Engineering student)

On the other hand, there are students who believe evaluation does not produce positive effects on the teachers’ performance, particularly when it is not complemented with a penalty system. Evaluation does not influence recruitment or test teachers and we would continue to observe a permanent reproduction of the same practices. Research is also overvalued and pedagogy is undervalued.

632 D. Leite et al. Constructive evaluation or alternatives to sanction A certain number of students portrayed evaluation in a clearly constructive perspective, in opposition to the perspective of evaluation as a penalty or sanction method. Evaluation is seen as a way to improve situations and to assume the shape of a permanent process; simultaneously, it should present itself as a whole and be concerned with the parts (to improve each degree) and should be directed to the development of the university. In this context, a student says:


Evaluation is always valid because it leads to reflection and generates the ability to reflect critically about the work of universities, their know-how and procedures crystallized by tradition, inducing the redefinition of goals and the establishment of new priorities. (Education student)

On the other hand, evaluation would be an exercise of social responsibility. Society or the community can evaluate universities on the quality of their teaching and on the services provided. It would then be important to know who evaluates, what criteria are used (production, dissemination, application of knowledge) and how the results are used (dissemination and result applicability). A student mentions: Evaluation is always important. The question is who evaluates, what the criteria are, its dissemination and result applicability. In the case of universities, evaluation should be done by all the surrounding community; the criteria should take into consideration the two main roles of universities, which are the production and dissemination of knowledge (is the university being effective?); finally the evaluation results should be widely broadcasted and applied. (Mathematics student)

In the perspective of some students, this would also be presented as an instrument of legitimacy for the restrictive financial policies of higher education: I believe evaluation should be a permanent, targeting enhancement. However, it seems to me that the government tends to minimise spending with education, using these evaluations as justification. In this sense, evaluation should be very complex and not simply a ‘is’ or a ‘is not’. A series of variables should be taken into consideration. Government should invest in the most precarious areas. There should be an increase in investment, anyhow. (History student)

State control and/or regulation of universities Specifically, in the Brazilian case, some students frequently identified institutional evaluation with the ‘Provão’ (‘Big Final’: the exam given to students at the end of their degree studies, whose results are used to evaluate Brazilian universities’ degrees) and they were politically sensitive to one of its more notorious consequences: publication of rankings based on the results of degrees and institutions. The use of the ‘Provão’ is a mechanism of control on the system and of higher education institutions by the ‘Evaluator State’ (Neave & Van Vught, 1994). This is done by direct intervention at the level of public universities’ financing and causes a certain number of indirect influences in the curricula and in the teaching/learning processes (homogeneity of curricula and teaching processes as a response to the ‘Provão’ format). In the opinion

The influence of institutional evaluation on universities 633 of the majority of students, the ‘Provão’ is a ‘vertical evaluation’, a top-down approach imposed by the ministry, which does not aim at the system’s improvement, but is used as an instrument to justify reducing financing of public universities: In my opinion, evaluation is the State’s form of control of what is being done in the degrees; they evaluate according to their own criteria and how is most convenient [for them]. I also believe they need to tell the people in charge of the country what our capacities are and to whom we can work for. (Education student)


The current evaluation type is not the most appropriate, as the ‘Provão’ is the key to MEC’s [Ministry of Education and Science] educational project, since the government has taken up the role of the institutional manager, threatening to close low quality public institutions, even if the responsibility for that lies with the government itself. (Chemistry student)

In an contrasting perspective, some students support this form of external evaluation, viewing it as a fairly positive procedure, such that it concerns the effects that can be induced in the system and institutions: it is an instrument used to improve teaching and allows for control of the situation as far as structure and quality of teaching goes, particularly because it identifies the universities that do not have structures to train good professionals: In my opinion, the MEC’s ‘Provão’ is valid, because at least the Education Ministry has a way to evaluate the private federal universities. (Veterinary student)

Some students generally attribute significance to evaluation referring to the role it can have in the regulation of institutions by the State: i.

There is the rejection of the homogeneity role—evaluation should encompass different university standards and should not be the same for every one of them; the evaluation criteria should be different, since their performance is not comparable: I think it is important to evaluate universities. However, I believe this evaluation should not be the same to every one of them. I think there should be criteria that distinguish this evaluation among universities. (Education student)

ii. On the other hand, several significant aspects of evaluation emerge as positive instruments of government intervention in the maintenance of teaching quality standardisation: I think this is positive in order to maintain an educational standard of quality of teaching/ learning, which has lately been forgotten by our government. (Education student)

Evaluation legitimacy The majority of the queried students perceive institutional assessment as a legitimate method of intervention in the running of universities. Evaluation brought improvements to the universities’ activities, concerning: i.

Students (wider concern with students; training for life; information about degrees; promotion of success);

634 D. Leite et al. ii. Teachers (improvement of teaching performance); iii. Curricula (curricula changes, adaptation of education to the job markets); iv. Education structuring (demand and quality, teaching methods, teaching efficiency); v. University procedures (creation of new degrees, image of the university, service improvement, influence on the decision making).


However, doubts are also cast on the method of developing the evaluation procedures. Even though evaluation is thought to allow an important diagnosis of the state of teaching (identifying failures, the strong and the weak points), some students mention the fact that it is not well linked to action, often not producing any practical effects. As two students (a Portuguese and a Brazilian) mention: I think that evaluation is important in all areas, including universities. However, I don’t think the method of assessment is very legitimate. (Tourism student) I believe that universities are being poorly evaluated, since the methods used do not match the reality of Brazilian universities. They can be evaluated, but only if you change the methods used nowadays. (Biology student)

In this context, it is assumed that evaluation would not have any sense or would not produce any effects in the organization and operation of universities: it does not generate any alterations to teachers’ practices or curricula, it does not increase performance, the criteria are not clear and it only exists in theory; teaching would be very difficult to judge and within the framework of the system it would undervalue the work of teachers. Furthermore, doubts are revealed about the ‘preventive’ role of evaluation in institutional changes and if it actually contributes to the improvement of the higher education system, in general, and to the adjustment of teaching methods to reality, in particular.

Institutional policies and aims of higher education In the students’ perspective, there is a set of assertions underlying institutional evaluation seen as the political tool of universities. In other words, evaluation is also faced as a policy to improve the institution’s operation, as well as to support the definition and achievement of the university and higher education goals and objectives. In any case, evaluation must emerge as a principle politically incorporated by the universities, giving voice to the majority as it should also be a way to inject optimism in the institutions, stimulating them to invest widely in the improvement of their performance: University evaluation should be the main goal of university policies. Politics is better when the voice of the majority is heard. Permanent improvement only exists with permanent auto-criticism, and the opposite is also true. In order to develop, we must think together! Union makes strength! Let there be more evaluations and let political decision be more the result of all the voices combined. (Mathematics student)

Some students’ approach establishes a straightforward connection between the roles of evaluation and higher education goals. Evaluation should have an objective

The influence of institutional evaluation on universities 635 direction so it does not deviate from its initial intentions. This direction seems to assume two sides: social relevance and economic performance. In the first case, which includes a wider vision of the universities’ social responsibilities, evaluation would be useful to level universities with societies’ aspirations and needs:


It is important [evaluation] because it is through evaluation that it will be possible to find out if universities are fulfilling their role in the face of the community. (Biology student)

As far as economic performance goes, sometimes in great proximity with the managerial phenomena, which overvalues the higher education economic goals, students understand evaluation as a correction instrument to equate university activities or their degrees to the job market. In this case, these aims are also directly linked to the social responsibility of the universities: (Evaluation)… is important because it enables improvement. Evaluation has a number of deficiencies and in order for them to be eased it should learn to listen and to respond to the needs of the job market and the society that sustains it. (Architecture student)

Evaluation and institutional comparison Some students, besides the fact of recognizing institutional evaluation as a way of diagnosing the real university conditions, are also aware that it may show the discrepancies existent among them, namely in terms of comparison of its degrees. The classification nature of evaluation creates a hierarchy of levels and differences among universities, displayed in the public domain. This phenomenon labels them and could damage the social image of the degrees. The rankings, in Brazil, resultant from this sort of evaluation (through ‘Provão’) and that supposedly, if you take the managerial approach, would serve information goals—knowledge of the comparative situation of universities as a way of facilitating their choices—and competition— ‘quality’ through competition—stigmatises undoubtedly the future professionals, whose degrees have had a less favourable assessment. It is important to undertake a periodical evaluation concerning our educational system, in order to improve it. I just do not agree with the way the broadcasting of the evaluation results is being done. The mentors of this assessment disregard the fact that if their degree is graded a C or a D they will not be able to compete in the job market with one that has had an A. What evaluators should be taking into consideration is that the dissemination of these results should not be done in this way, since they are prematurely excluding possibly great professionals that were just unfortunate to study in a college with few resources, but that individually show great potential. (Agronomy student)

Conclusions Throughout this research, we were able to observe that students’ perception of evaluation is quite diversified. It covers a vast area that refers to distinctive features of higher education dimensions: government policies, the system’s operation, institutional policies, curricula and the organization of teaching and learning process. The


636 D. Leite et al. students’ perceptions are divergent and consistency in their approach, at the individual level, is also difficult to detect. The approach to evaluation is, most of the times, polarised between positive and negative extremes, frequently assuming hybrid configurations. This phenomenon clearly demonstrates that it is not exactly a reproduction of the various ‘external’ approaches given to the students: political, institutional and educational. It emerges with original items that integrate critical elements and true reorientation proposals to the creation and development of institutional evaluation in the universities. If we accept that the students’ perceptions on the evaluation of universities are collective, developed and shared creations, at various levels, it is then possible to argue that its analysis is quite appropriate in order to understand their modus operandi in and upon the institution. They indicate an effort to adapt to the institutional environment as well as a demonstration of the will to influence it. Thus, these perceptions may be valid as a reality of the institutional evaluation itself. It is not an objective reality, but common, subjective views that emerge as a shared reality: through the evaluation of universities students recognize and share somewhat their own experiences and the established institutional, educational and political conditions, but are also able to act upon them. Notes on contributors Rui A. Santiago is Director of the Master in Higher Education Policy and Management, University of Aveiro, and a researcher with the Centre for Research in Higher Education Policies, Portuguese Universities’ Foundation, Portugal. Cláudia S. Sarrico is Director of the degree in Public Administration, and a member of the scientific board of the Master in Public Management, University of Aveiro, Portugal. Denise Leite is Professor of the Postgraduate Programme in Institutional Evaluation, Federal University of the Rio Grande do Sul, Brazil. Cecília Loréa Leite is a lecturer at the Federal University of Pelotas, Brazil. Marlis Polidori is a researcher with the Office of Institutional Evaluation, Federal University of Pelotas, Brazil. References Barnett, R. (1990) The idea of higher education (Buckingham, The Society for Research into Higher Education & Open University Press). Barnett, R. (2000) Reconfiguring the university, in: P. Scott (Ed.) Higher education re-formed (London, Falmer Press). Biggs, J. (1987) Student approaches to learning and studying (Melbourne, Australian Council for Educational Research). Bos, W. & Tarnai, C. (1999) University faculty members’ and students’ perceptions of university academic systems, International Journal of Educational Research, 31(8), 699–715. De Boer, H. (2003) Who’s afraid of red, yellow and blue? The colorful world of management reforms, in: A. Amaral, V. L. Meek & I. M. Larsen (Eds) The higher education managerial revolution? (Dordrecht, Kluwer).


The influence of institutional evaluation on universities 637 Entwistle, N. & Ramsden, P. (1983) Understanding student learning (London, CromHelm). Green, D. (1994) What is quality in higher education? Concepts, policy and practice, in: D. Green (Ed.) What is quality in higher education? (Buckingham, The Society for Research into Higher Education & Open University Press). Harvey, L. & Green, D. (1993) Defining quality, Assessment and Evaluation in Higher Education, 18(1), 9–34. Jenkins, A., Blackman, T., Lindsay, R. & Patou-Saltzberg, R. (1998) Teaching and research: student perspectives and policy implications, Studies in Higher Education, 23(2), 127–141. Leckey, J. & Neill, N. (2001) Quantifying quality: the importance of student feedback, Quality in Higher Education, 7(1), 19–32. Leite, D. (1990) Aprendizagem e Consciência Social na Universidade (Doctoral Thesis, University of the Rio Grande do Sul, Brazil). Leite, D. (1992) Metacognição e aprendizagem na universidade, Revista de Educação, VII(2), 55–64. Leite, D. (2002) Avaliação institucional, reformas e redesenho capitalista das universidades, Revista Avaliação, 7(2), 29–48. Lizzio, A., Wilson, K. & Simons, R. (2002) University students’ perceptions of the learning environment and academic outcomes: implications for theory and practice, Studies in Higher Education, 27(1), 27–52. Massen, P. (2003) Shifts in governance arrangements: an interpretation of the introduction of new management structures, in: A. Amaral, V. L. Meek & I. M. Larsen (Eds) The higher education managerial revolution? (Dordrecht, Kluwer). Meek, L. V. (2002) On the road to mediocrity? Governance and management of Australian higher education in the market place, in: A. Amaral, G. A. Jones & B. Karseth (Eds) Governing higher education: national perspectives on institutional governance (Dordrecht, Kluwer). Meek, L. V. (2003) Governance and management of Australian higher education: enemies within and without, in: A. Amaral, V. L. Meek & I. M. Larsen (Eds) The higher education managerial revolution? (Dordrecht, Kluwer). Miller, H. D. R. (1995) The management of changes in universities, Sstate and economy in Australia, Canada and UK (Buckingham, The Society for Research into Higher Education & Open University Press). Morosini, M. C. & Leite, D. (1997) Avaliação institucional como um organizador qualificado, in: V. Sguissardi (Ed.) Avaliação Universitária em Questão: Reformas do Estado e da Educação Superior (Campinas SP, Ed. Autores Associados). Neave, G. & Van Vught, F. A. (1994) Conclusions, in: G. Neave & F. A. Van Vught (Eds) Prometheus bound, the changing relationship between government and higher education in Western Europe (Oxford, Pergamon Press). Powell, A. M.; Hunt, A. & Irving, A. (1997) Evaluation of courses by whole student cohorts: a case study, Assessment & Evaluation in Higher Education, 22(4), 397–404. Ramdsen, P. (1985) Student learning research: retrospect and prospect, Higher Education Research and Development, 4(1), 51–69. Reed, M. I. (2002) New managerialism, professional power and organizational governance in UK universities: a review and assessment, in: A. Amaral, G. A. Jones & B. Karseth (Eds) Governing higher education: national perspectives on institutional governance (Dordrecht, Kluwer). Reynolds, M. & Trehan, K. (2000) Assessment: a critical perspective, Studies in Higher Education, 25(3), 267–278. Rousseau, R., Potvin, P., Papillon, S., Trembley, Y., Santiago, R. A., Loscertales, F., Oliveira, L., Terça, O., Nuñez, L. & Guil, A. (1995) Perception du professeur d’université en éducation: étude interculturelle, Canadian and International Education, 22(2), 17–50. Salminen, A. (2003) New public management and Finish public sector organisations: the case of universities, in: A. Amaral, V. L. Meek & I. M. Larsen, The higher education managerial revolution? (Dordrecht, Kluwer).


638 D. Leite et al. Sander, P., Stevenson, K., King, M. & Coates, D. (2000) University students’ expectations of teaching, Studies in Higher Education, 25(3), 309–324. Santhaman, E. & Hicks, O. (2002) Disciplinary, gender and course gear influences on students’ perceptions of teaching: explorations and implications, Teaching in Higher Education, 7(1), 17–32. Santiago, R. A. & Carvalho, T. (2004) Effects of managerialism on the perceptions of Higher Education in Portugal, Higher Education Policy, 17, 427–444. Santiago, R. A., Leite, D., Leite, C. & Sarrico, C. (2003) Avaliação institucional e aprendizagem organizacional nas universidades: os dilemas da adaptação ou da reconstrução sob pressão do ‘managerialismo’, Revista Portuguesa de Administração e Políticas Públicas, III(1/2), 2–13. Santiago, R. A., Tavares, J., Taveira, M. C., Lencastre, L. & Gonçalves, F. (2001) Promover o sucesso académico através da avaliação e intervenção na universidade, Avaliação, Revista da Rede de Avaliação Institucional da Educação Superior, 6(3), 31–43. Tavares, J., Santiago, R. A. & Lencastre, L. (1998) Insucesso no 1° ano do Ensino Superior: um Estudo no Âmbito dos Cursos de Licenciatura em Ciências e Engenharia na Universidade de Aveiro (Aveiro, Universidade de Aveiro).