THE EUROPEAN ADDED VALUE OF FRAMEWORK PROGRAMMES: EVIDENCE FROM THE UK Alessandro Muscioα JEL Classification: G18, O16, O33, O38 Keywords: EU Framework Programmes, European Added Value, R&D
1. Introduction There is a strong support in the economic literature for the idea that collaborative programmes represent a valid policy instrument in reducing the uncertainties, the risks and the costs deriving from undertaking advanced technological projects. Over the last twenty years government support to the establishment of public-private R&D partnerships has been at the forefront of research and innovation policy (Dodgson, 1993). Indeed, government funding may reduce firms’ costs and obstacles in establishing R&D collaborations and access to complementary types of external expertise (Coombs et al., 1996; Branscomb and Keller, 1998; Edquist, 1997; Katz and Martin, 1997). In return, collaborations may facilitate the diffusion of economically relevant knowledge within the private sector, increasing the impact of public investments (Narula and Dunning, 1998). With the creation of Framework Programmes the relevance of the international dimension of collaborative research has greatly increased (Kuhlmann and Edler, 2003). The multi-annual Framework Programmes (FPs) represent the medium-term planning instrument for Research and Technological Development (RTD) at EU level. However, the creation of international research programmes such as FPs, has also raised a number of questions about the benefits deriving from research conducted at the international level (Luukkonen, 1998). The rationale behind international research programmes is that they are supposed to ‘add value’ which cannot be obtained through national programmes (Fahrenkrog, 2002; Georghiou 2001; Georghiou et al., 2002; PREST et al., 2002). Multilateral programmes are perceived as having higher transaction costs deriving from management: the question is whether these The author would like to thank the staff of Technopolis Limited, Brighton, for their support. Dipartimento di Scienze Economiche e Aziendali (DPTEA) – Università Luiss Guido Carli, Via O. Tommasini, 1 - 00162 Roma (Italy), Tel: + 39 06 86506530, Email:
[email protected] α
381
additional costs are offset by the additional value associated with the international dimension. However, few research works have questioned the wider impact of FP collaborations and the additionality of frameworks with respect to national research programmes (Luukkonen, 2000; SQW Ltd, 1999). According to Arnold et al.: “FP programmes […] tend to play a distinct role in the European Research and Innovation System, and do not simply duplicate national schemes” (Arnold et al. (2005: forthcoming). Indeed, the most obvious addition is the ability to tackle problems on the basis of drawing together a much larger pool of resources, both financial and human. Some research problems may involve such costly equipment as to be economically feasible only with international cooperation, as in branches of nuclear physics or space. Others may benefit from ‘economies of scale’ such that increasing returns are obtained from larger inputs of resources (e.g. bioinformatics). In addition to the normal ‘static’ economies of scale, the recent theoretical literature has highlighted ‘dynamic’ economies of scale (von Tunzelmann et al., 2003). The latter include the ability to achieve outcomes faster by scaling up the projects, and indeed one of the major justifications for international collaboration is the desire to achieve tangible results more quickly, for example, through subdividing tasks to permit parallel working of research teams. The idea that international collaboration may work as an accelerant is particularly attractive when the overall programme objective is concerned with boosting ‘competitiveness’, since competitive products in hightechnology industries demand speed to market. Another branch of the recent business literature has come to emphasise economies of ‘scope’ as well as ‘scale’ (von Tunzelmann et al., 2004). In this respect, international programmes offer the opportunity to attack problems in different ways, stemming from the varieties of research cultures of the participants, and to attain different forms of outcome. The literature on innovation suggests that variety – and competition among competing studies and approaches – may be essential for eventual success. National targeted research programmes often commit to one particular line of development (‘path dependency’), which may turn out to be fruitless or uncompetitive. Given the high degrees of uncertainty attached to any one course of action, the opportunity to operate with a more diverse portfolio, as provided by international programmes, may be seen as an important safeguard as well as being a healthy source of cross-fertilisation and rivalry. At the same time, there is a theoretical risk that the attraction of economies of both scale and scope may lead administrations to venture too 382
far down that particular road to the point where such programmes could impede rather than expedite progress. A long tradition in economics, dating back to the classic works of Alfred Marshall (1890), has argued that, while physical economies of scale in production might go on rising as volume increases, at some point managerial ‘diseconomies’ of scale are likely to arise. In a research context, for instance, those diseconomies may reveal themselves in networks that have grown so large as to have become ungovernable, and to have compromised their very reason for existence. This discussion of both economies and diseconomies of ‘scale and scope’ fits seamlessly into policy-makers’ concerns about ‘additionality’ and ‘added value’. The decision to enlist in – or in someway support – an international research programme can be seen as a desire on the part of national officials to do ‘more’ or ‘different’ in order to do ‘better’. It will also emerge that the R&D will not be done better if diseconomies of scale or scope take over. A key issue is the integration and cohesion of the activities these international programmes involve, and that necessitates effective administration. 2. FPs and European added value 2.1 The concept of European Added Value (EAV) The economic literature is consistent with regard to the unique potential of an international research and development programme, as compared with (even large) national programmes (Pavitt, 1998). The arguments for RTD at a European scale, as opposed to a national scale, are set out in several recent FP evaluation reports, including those by Yellow Window et al. (2000) and PREST et al. (2002). Throughout the last two decades each new revision of Framework has marked important changes in European research policy.1 The evolution of FPs reflects the debates and overall evolutionary processes concerning EU institutions and transfers of powers and policy instruments from Member
1
Throughout different revisions of Frameworks preference was indicated for research that was: • conducted on a large scale, for which single Member States cannot provide the necessary financial means and personnel, or which can benefit financially from being carried out jointly • complementary to national programmes • addressing common interests, and could be best tackled through joint effort • likely to contribute to the cohesion of the common market, promoting the unification of European science and technology, and contributing to the establishment of uniform laws and standards
383
States to European bodies. At the same time the FP is a small fraction of EU state R&D support. In the first revisions, the primary objective was to increase the competitiveness of European industries (not individual participating companies) by improving their technological capabilities. The intention was to bridge the perceived technological gap between Europe on the one hand, and the US and Japan on the other. A further motivating factor was the perceived excellence of Europe in the quality of its basic science, but its apparent inability to translate this into commercially successful products (Luukkonen, 2000). From the outset, issues of scale and scope were placed at the forefront, but reined in by intentions of integration and cohesion as seen from a Europeanwide standpoint. In following revisions of FPs competitiveness has remained the primary goal of European technology policy, but a wider set of objectives has been added over time, such as reducing unemployment, accelerating structural changes, and ensuring greater cohesion. Accordingly, successive Framework Programmes have addressed wider areas of intervention. In particular, the latest FPs have shifted the emphasis from supply-side factors, central in the design of the first policies, to include diffusion-oriented actions and the increase of learning skills and knowledge among Europeans. FP5 (1998-2002) marked important changes in the EU approach to the goals of collaborative research. FP5 put a greater emphasis on the European Added Value (EAV) of research carried out at the international level in Europe. EAV is clearly a complicated phenomenon. As suggested by PREST et al.: “The concept of EAV is actually a composite of a number of many other tricky concepts, few of which are widely understood and none of which are easy to measure. These include the ‘EU concepts’ of subsidiarity and proportionality, the related concept of additionality, in all its various forms, and even the generic – but still difficult to define – concept of value itself” (PREST et al., 2002:114). International – and European-level – collaborations undertaken under Framework may be expected to offer benefit over national programmes for several reasons: Critical mass. A ‘critical mass’ of research performers – and users – is unlikely to be achieved nationally, or where a national focus would exclude ‘key players’ from the joint venture. Framework provides an opportunity to tackle research questions at a scale that few EU member states would contemplate or could afford; 384
European Issues. A mechanism through which to tackle specifically European issues, in particular those relating to EU policy. Much of this work underpins the development of the EU itself, looking at issues such as EU enlargement, social cohesion, employment, movement of people, and would be unlikely to be supported by any member states; • Costs. Research projects can involve massive capital expenditure, where no one member state could be expected to underwrite the entire cost, or where the duplication of effort or facilities would make no sense economically or technologically; • Gap filling. In certain areas, Framework provides an opportunity to seek support for given kinds of activity that is missing or features less prominently in National policy repertoires; • Standards. An international consensus among a broad spectrum of producers and users that a given scientific or technical question is needed to secure progress, and for mutual benefit, as with the determination of the scientific foundations of international telecommunications standards, which in turn facilitate the more efficient operation of global markets; • Enter new Markets. Companies wish to extend their international supply-chains and enter new foreign markets. There are obviously close trading links between EU countries and innumerable supply chains involving companies from more than one country, and international technology projects can facilitate closer interaction; • Commonality. A trans-national interest exists in several research areas, such as in the area of environmental pollution or European social cohesion. Framework offers a mechanism through which to tackle common issues, such as climate change, which will yield only to substantial investment over extended periods of time; • Multilateralism. A mechanism through which to access skills, knowledge and facilities not available nationally. Moreover, ‘multicountry’ studies often hold the promise of more reliable and more applicable results than do national ones. The concept of EAV was given prominence in FP5, where it was explicitly adopted as a key requirement to be considered in the project selection process, following the 1997 Five-Year Assessment of the Framework Programmes (EC, 1997). The particular ‘value added’ criteria to be adopted were the need for ‘critical mass’, a requirement for a significant contribution to the implementation of one or more Community policies, and relevance to Community problems, standards or development of the European area. Other groups of criteria for FP5 relate to social objectives and to economic development and scientific and technological prospects. Overall, 385
the groups of criteria have perhaps fairly been criticised for juxtaposing different categories of criteria (e.g. social/economic criteria on the one hand, EAV on the other) and treating them as essentially independent, where previously EAV was treated as a necessary condition for attainment of other goals (Yellow Window et al., 2000). Nevertheless, there is also some defence for broaching these wider criteria for assessing EAV. The rationale for public direction of and control over technology has a multitude of aspects that spread across social, economic, political and organisational as well as technological issues (von Tunzelmann, 2004). While many of these are difficult to pin down in terms of economic values, it seems appropriate to relate the outcomes to externally given desiderata in the form of policy objectives, even (or especially) when those objectives stem from quite different sources. Technology itself inescapably brings consequences in many of these arenas, many of which are well known (e.g. employment, inequality, exclusion) although little attempt has been made to draw the ‘big picture’, i.e. the totality of ‘third-order’ impacts. However, despite the overall Barcelona objectives of substantially raising levels of total R&D, at present EU governments’ funding invested in FPs is fairly low (Soete, 2005). This relatively low’ proportion of public R&D invested in FPs is thought to limit their overall impact. But this would be increased if their major effect were to act as an important trigger for further private R&D. Notwithstanding measurement difficulties, their ‘added value’ could be considered according to this criterion. Assuming the firm is rational, it would not undertake the additional investment unless it considered it profitable, so there are no ‘moral hazard’ issues. And problems of attribution attached to commercial outcomes, and particularly concerns about additionality and nearness-to-market which can be raised by rapid commercial success, are avoided. This applies to both national and international programmes, in fact the distinction between the two could rest ‘purely’ on the question of which format is likely to lead to the most additional research, or (using the EC terminology) to the extent of ‘input additionality’. According to Luukkonen we can define additionality as: “…the difference which government-sponsored programmes have made to the recipients, particularly companies, in terms of R&D activities.” (Luukkonen, 2000:711) When firms invest heavily in additional related research, it is better evidence of value and commitment than any subjective assessments. European-level research would be preferred if, through for example, better networks or more highly trained researchers, it led to more research in the long run. 386
2.2 Research Hypothesis Evaluation of Frameworks is carried out at the country level by independent organisations. Therefore in large measure conclusions on achievement of goals and evaluations of positive and negative aspects of participation in FPs must be drawn from the analysis of country-specific studies (see for example: Graversen and Siune, 2000; ISG, 2001; NIFUSTEP and Technopolis, 2004; Technopolis, 2001, 2004; Schibany et al, 2001; Uotila, 2004). The aim of this paper is to address the issue of EAV through the examination of UK participation in FP4 and FP5. The UK represents a good testbed for evaluating the EAV of frameworks. It is one of the largest EU members, it has a competitive research community and a strong business base, with a fairly balanced mix of both SMEs and large firms. The UK context provides a sufficiently large population of potential participants and allows investigating differences in impacts of participations on different types of organisations. In the case of the UK, performance in Framework has been extremely strong. In FP4 and FP5 the UK participated in and coordinated more projects than any other country. Performance by the UK Higher Education Institutions (HEI) sector has been especially exceptional, with UK universities securing almost a quarter of all FP5 contracts placed with all countries’ HEIs and half of all funding allocated to UK participants. We investigate whether conducting research at the European level has beneficial effects on FP participants and assess the effectiveness of FPs in delivering EAV. The study will focus on EC data on UK participation into Frameworks and on the results of a questionnaire survey targeting participants in framework projects carried out by Technopolis for the UK Department of Trade & Industry (DTI) (Technopolis, 2004). Outcomes of interviews with UK policy makers directly involved into Framework research are also considered. 3. UK performance in framework 3.1 Introduction This study is focused on UK involvement in, and the impacts arising from, projects undertaken within FP4 and FP5, which operated from 19941998 and 1998-2002 respectively. However, some elements of the study relating to programme design and administration have also illuminated the current programme – FP6 – which began in 2002 and is due to conclude its operations in 2006. 387
The study methodology involved the analysis of data from different sources. Several discrete but related packages of work were carried out: 1. Analyses of European Commission (EC) data on UK participation in the two most recently completed programmes (FP4 and FP5) and preliminary data on UK involvement in the current programme (FP6); Data on UK participation in FP projects were provided to the Office of Science and Technology (OST) by the European Commission; 2. A series of semi-structured interviews with 34 UK Programme Management Committee (PMC) members, National Contact Points (NCPs) and other officials with responsibility for, and oversight of, various aspects of UK involvement in the Framework Programmes; 3. A questionnaire survey directed to all UK participants in FP4 and FP5. Two questionnaires were used to gather (i) participants’ general views on the Framework Programmes and the arrangements for FP6, and (ii) information on the outputs and impacts arising from individual FP4 and FP5 projects. Overall, we collected a total number of 2,701 completed questionnaires in our survey. Both interviews with government representatives and questionnaires to FP participants dealt with several issues related to EAV of research sponsored by frameworks. Based on these data sources, the following sections analyse, in order: UK participation via analysis of UK participation in the two most recently completed Framework Programmes (FP4 and FP5); goal attainment in framework projects; EAV of framework research. 3.2 UK participation into Frameworks Framework allocates funding on a competitive basis. This means that statistics and trends regarding UK proposal success rates, coordination, participations and Framework income are good indicators of UK ‘performance’, as compared with the performance of other EU member states and particularly the larger countries and leading scientific nations. It is assumed that stronger performance on these measures will have a positive impact on the scale and share of benefits derived. UK participation into frameworks is very strong. In FP4, UK organisations took part in 47% of total projects funded (7,276 out of 15,457).2 This was the highest share of any member state, and significantly 2
Data supplied by DG Research showed that there were a total of 24,664 FP5 proposals with UK partners and that 6,625 of these were funded, giving an overall success rate of
388
more than countries such as Germany or France, which participated in 41% and 38% projects respectively. In FP5, UK organisations were involved in 41% of total projects funded (6,613 out of 16,251). Once again, this level of involvement was the highest of any member state and more than both Germany and France, which participated in 40% and 34% of projects respectively. This decline in project participation from FP4 to FP5 is not unique to the UK. Growth in the participation of the Accession Countries, and to a lesser extent third countries, such as Russia and Canada, has contributed to a general reduction in levels of involvement for all Member States, however in proportionate terms the UK registered the strongest drop. The average share of project involvement for all member states fell by 6%, while the equivalent figure for the UK fell by 14%.3 Under FP5, UK organisations were awarded EC funding totalling €2,047 million (or 15.9% of the total), which is very much in line with UK share of EU GNP. The costs of the work eligible for Framework funding undertaken by UK partners within FP5 projects totalled €2,965 million, so the FP contributed 69% of UK partners’ total eligible costs. The remainder – some €918 million – was met by participants themselves, or from UK public sector funding. The average amount of funding obtained by UK organisations per participation was €188k, 17% above the average for FP5 overall (€160k per participation). Table 1 uses DG Research data to show UK participation in FP5 by type of organisation. In particular it aims to show how different categories of UK participant have performed in comparison with each other and in comparison to other countries. In the UK by far the greatest share of participations was undertaken by the higher education sector, with almost half of all UK FP5 participations. UK higher education establishments accounted for a far higher share than that obtained by European HEIs overall, confirming the dominant performance of the UK HE sector, not just in comparison with other types of organisation in the UK but as compared with HEIs in other EU member states. At 27%, the share of UK enterprise participations was slightly below the EU average.
3
27%. This success rate compares well with those seen in UK national collaborative research schemes, such as LINK. Under the early calls of FP6, UK organisations were involved in 53% of the proposals retained for funding, which suggests that the downward trend in UK participation may have been reversed. However, the new FP6 instruments (IPs and NoEs) tend to involve larger numbers of partners and countries and, therefore, one might expect project participation rates to increase for all countries when compared with earlier Framework Programmes.
389
Table 1 - UK participation in FP5, by organisation type Organisation participations Type - all countries Higher Education 24,646 establishments Enterprise 23,094 sector Public research 13,788 centres Public (non research) 6,412 sector Private nonprofit research 5,072 centres Private commercial 2,751 research centres Private non2,604 profit Not defined 1,701 Total 80,068 Source – DG Research
%
participations - UK only
%
UK Share of participations
31%
5,066
46%
21%
29%
2,926
27%
13%
17%
934
9%
7%
8%
488
4%
8%
6%
560
5%
11%
3%
450
4%
16%
3%
298
3%
11%
2% 100%
183 10,905
2% 100%
11% 14%
3.3 Project outputs Our questionnaire survey (1,275 responses) provided the opportunity to investigate the nature and extent of Framework outputs. This analysis aims at providing an important check on the programme’s rationale and added value. Our analysis shows that Framework projects produce an average of 25 outputs each, and that roughly 10% of the projects have produced over 100 outputs. The primary outputs, in numerical terms at least, are publications, with an average ‘per project’ output of almost 10. The majority (79%) of the projects have produced at least one peer-reviewed publication and around 10% have produced more than 20 such outputs. Non peer-reviewed publications were the next most frequent type of output, with a ‘per project’ output of almost seven. Over two-thirds of the projects (67%) have produced at least one ‘non-peer-reviewed’ publication.
390
The next most common type of output was new or improved tools, methods and techniques, with two-thirds (67%) of the projects producing at least one such output and an overall average of two per project. Just under half of the projects (46%) have produced newly qualified personnel (trained Masterslevel or PhD-level graduates). By comparison, a much lower proportion of projects produce new or improved commercial products and services or new or improved scientific or industrial processes. Less than a third of the projects have produced at least one of each of these types of output. Patent applications are uncommon, with only one in six of the participants indicating that their project had resulted in an application being filed. Participants were asked to indicate the importance of each type of output to their own organisation by ranking their ‘top five’ where 1 was the most important. Table 2 illustrates the results. Table 2 - Participants’ views on the importance of different types of output (n=1,261)
Publications in refereed journals and books New or improved tools, methods or techniques Other publications Newly qualified personnel (e.g. MSc, PhD) New or improved commercial products or services New or improved scientific or industrial processes New or improved facilities or infrastructure New or improved technical codes or standards Patent applications New or improved regulations or policies
Share of participants ranking 1-5 79% 74% 57% 45%
Share of participants ranking 1 37% 24% 6% 5%
34%
11%
34% 21% 20% 18% 14%
6% 4% 5% 3% 3%
There were evident differences in the rankings by the public and private sectors, on most kinds of output: New or improved commercial products or services - almost twothirds (64%) of the private companies rated these as important outputs as compared to just one in five (20%) of respondents from the public sector research base 391
Refereed publications – almost all (94%) of the public sector researchers rated these as important outputs as compared to 57% of private companies Newly qualified personnel – over half (59%) of the public sector researchers rated these as important outputs from their Framework projects, but less than a quarter (23%) of the private companies did the same The most significant difference was found in relation to participants’ ratings of peer-reviewed publications, with the universities and public research institutes rating these far more highly than companies and private research institutes. The next strongest difference related to new or improved commercial products or services: companies were more likely to value this type of output, whilst the universities and public research institutes tended to not rate these as important. 3.4 Goals and benefits 3.4.1 Participants’ goals The questionnaire sent to FPs participants explored their motives for becoming involved in Framework. Table 3 shows, for each motive, the proportion of respondents that (i) ranked that motive as one of the top five reasons behind their involvement (ranked 1-5), and (ii) ranked that motive as their primary motive (ranked 1). Gaining access to research funding was one of the top-five reasons for involvement for 84% of respondents, and was cited as the primary motive for almost a third (30%). This finding serves to remind us that most Framework participants are, first and foremost, research performers that need to secure funding in order to operate. Whatever their other motives may be, gaining access to research funding is critical for many of the organisations that take part in Framework.
392
Table 3 - Motives behind involvement in Framework projects (n=1,263)
Motive To access research funding To develop and extend internal knowledge and capabilities To develop new or improved relationships or networks To get answers to specific scientific or technical questions To develop new or improved tools, methods or techniques To develop new or improved commercial products or services To develop new or improved scientific or industrial processes To create new or improved facilities or infrastructure To develop new or improved technical codes or standards To develop new or improved regulations or policies Other
Share of participants giving a 1-5 ranking 84%
Share of participants giving a 1 ranking 30%
82%
11%
79%
9%
78%
16%
73%
11%
34%
7%
31% 24%
4% 4%
23% 21% 8%
3% 4% 2%
Having accessed research funding, participants are then most interested in the ‘softer’ types of knowledge and networking benefits, and are relatively less concerned about the more concrete types of ‘output’ that might arise from the projects. The fact that the development of new or improved commercial products or services was only cited as the primary motive for around one in fourteen (7%) of our respondents, and figured in the top-five for only one-third (34%), serves to further underline the fact that Framework participants are not, for the most part, expecting commercial impacts to flow directly from projects. The findings presented in Table 3 help to further confirm that, at its heart, Framework is a mechanism that provides research performers with funding to allow them to tackle specific scientific and technological issues or problems, extend their knowledge and capabilities and maintain international networks. For each category of organisation, Table 4 presents the two motives that were cited as the primary reasons for involvement by the greatest proportion of respondents within that group. Taking large companies first we see that the development of new or improved tools, methods or techniques and getting answers to specific scientific or technical questions were the motives ranked 393
first by the greatest share of respondents. The motives cited as most important by most SMEs were the development of new or improved commercial products or services and gaining access to research funding. For all other categories of organisation, the motives ranked as most important by the largest share of participants were gaining access to research funding and getting answers to specific scientific or technical questions. These findings make sense intuitively. Large companies are less likely than other types of organisation to need to gain access to research funding in order to perform research, so this motive figures as less important for them. Instead, the actual project results or concrete developments, embodied in tools, methods and techniques, are the items most sought. In contrast, SMEs are focused on their short-term needs – to gain access to funding for research that will, in turn, lead to the development of new or improved commercial products or services. Finally, the public sector research performers (universities and research institutes) tend to focus on accessing funding in order to drive forward the status of knowledge in a particular area. Table 4 - Primary motives behind involvement in Framework projects, split by type of participant (n=1,263) Motives ranked 1st by highest proportion of respondents To develop new / improved tools, methods or techniques (17%) Large companies To get answers to specific scientific or technical questions (16%) To develop new / improved commercial products or services (17%) SMEs To access research funding (15%) To access research funding (37%) Private research To get answers to specific scientific or technical questions institutes (13%) To access research funding (39%) Universities/colleges To get answers to specific scientific or technical questions (19%) To access research funding (37%) Public research institutes To get answers to specific scientific or technical questions (21%) To access research funding (18%) Others To get answers to specific scientific or technical questions (16%)
Organisation type
394
3.4.2 Goal attainment Respondents’ own rankings as to which factors were most important in motivating them to take part in their project were re-coded such that (i) items ranked as being a top-five motive were taken to be highly important ‘objectives’ for their participation, and (ii) items that were not ranked in the top five were considered to be of low importance for their participation. We then classified respondents as having high levels of achievement with respect to each objective if (i) their project had produced at least one output of that type (e.g. one or more new or improved tools, methods or techniques), or (ii) the respondent had reported that the impact of the project was medium-high in that respect (e.g. medium or high impact rating in terms of improved relationships and networks). Otherwise, respondents were considered to have realised only low levels of achievement with respect to that objective. Table 5 shows that different categories of participants experience different levels of goal attainment, although, reassuringly, all participant categories achieve over 60% of the goals they ranked as important. Table 5 - Levels of attainment of ‘high importance’ objectives, by type of organisation (n=5,723 [objectives])
Public research institutes Private research institutes Universities/colleges SMEs Large companies Others
Number of objectives rated as important 661 239 2718 1194 664 247
Proportion of cases where attainment is high 84% 82% 80% 74% 71% 61%
The data were re-analysed separately for each ‘objective’ in order to establish whether Framework is more or less successful at meeting certain types of objectives. Table 6 shows for each objective the number of respondents that rated it as of high importance; and the proportions of cases in which the attainment of that objective can be considered to be high and low.
395
Table 6 - Levels of attainment of ‘high importance’ objectives, by type of objective (n=5,723 [objectives]) Number of respondents Objective rating the objective as important To access research funding 1067 To develop new or improved relationships or networks 972 To get answers to specific scientific or technical questions 955 To develop and extend internal knowledge and capabilities 1005 To develop new or improved tools, methods or techniques 766 To develop new or improved commercial products or 343 services To develop new or improved scientific or industrial processes 321 To develop new or improved technical codes or standards 234 To create new or improved facilities or infrastructure 250 To develop new or improved regulations or policies 209
Proportion of cases where attainment is high 100% 87% 86% 82% 73% 54% 49% 40% 38% 22%
The first objective – to access research funding – can be considered to have been universally attained since all of the projects on which the respondents are providing feedback were funded through Framework. The next most highly achieved objectives are the development of new or improved relationships or networks, gaining answers to specific scientific or technical questions, and the development of internal knowledge and capabilities. Where respondents indicated that these were important objectives for their participation in Framework, over 80% also indicated that they had achieved medium-high levels of impact in these terms. This suggests that Framework works well in respect of its core functions – promoting research collaboration across Europe and strengthening S&T knowledge and capabilities. The more specific objectives are realised in a lower proportion of cases. Attainment levels were lowest for technical codes & standards, facilities & infrastructure and regulations or policies, though these types of benefit tend to take a long time to be established so attainment levels may rise in the future.
396
Taken together, the findings suggest that Framework is better at delivering the ‘softer’ knowledge and networking benefits sought most by the universities and public research institutes, and is least able to deliver the more concrete, commercially-oriented outputs and benefits sought by companies, both large and small. 3.4.3 Impacts of Framework projects on participants Participants were presented with a range of potential benefits and asked to rate the level of impact attained on each, for their own organisation. Figure 1 shows that the primary ‘direct’ impacts of Framework projects are that they improve the knowledge base of the participants, both in general terms and in relation to specific issues, and that they improve the extent to which organisations are networked and ‘respected’ on the international stage. By comparison, participants are less likely to report high impacts in terms of the more ‘commercial’ types of benefit. Universities/colleges and public sector research institutes were significantly more likely than other types of participant to report high levels of impacts on their organisation in terms of most of the listed categories of benefit. In contrast, businesses reported significantly lower levels of impact overall. The difference was most marked in terms of improved relationships and networks, enhanced reputation and image, and improved competitive position internationally. In addition, large firms were significantly less likely to report a high impact in terms of their improved competitive position, and SMEs and private research institutes were significantly less likely to report an improved ability or capacity to conduct R&D.
397
Figure 1 - Participants’ views on the nature and extent of project benefits realised by their own organisation (n=1,243) No impact
Low impact
Medium impact
High impact
Improved understanding of specific issues/problems Improved relationships and networks Enhanced reputation and image Improved internal knowledge and capabilities Improved competitive position internationally Improved competitive position nationally Improved ability or capacity to conduct R&D Improved turnover or market share 0%
20% 40% 60% 80% Percentage of respondents
100%
3.5 European added value 3.5.1 Introduction The evidence presented so far has shown the extent of UK participation into frameworks and primary goals and benefits deriving from it. UK organisations achieve very high levels of important objectives from participating in international collaborative projects and frameworks seem to have a substantial impact in a number of areas related to knowledge development. With Frameworks the EC has been trying to support research of an European dimension, starting from the assumption that this super-national approach, besides stimulating cohesion between different research systems and networking between European institutions, may also provide ‘value added’ not available otherwise at the national level. This idea of EAV represents the key rationale of frameworks and can be investigated in terms of worthwhile outcomes on the European level that would not occur otherwise. The aim of this section is to address the issue of EAV of frameworks. Indeed, a relevant part of our questionnaire survey aimed at investigating the issue of EAV of framework research. The EAV of frameworks was primarily determined enquiring participants about the motives for undertaking projects 398
in collaboration with international partners, and about the reasons why such projects could not have been funded and undertaken at the national level. Complementarily, we report here the results of interviews to UK policy makers directly involved into Frameworks. Interviewees provided useful qualitative information coming from different viewpoints on the importance of the European dimension of research. 3.5.2 EAV in the view of government representatives As part of our FP5 evaluation we carried out a series of semi-structured interviews with UK programme management committee members (PMCs) and members of the national contact point (NCPs) teams. In total we interviewed 34 UK PMCs and NCPs, often in teams. Our programme of interviews covered the majority of FP6 thematic areas and specific programmes, including nuclear research (Euratom). Table 7 reports a breakdown interviews undertaken by FP6 priority area. Overall our interviewees were primarily representatives of UK Research Councils and government departments. Table 7 - Breakdown of institutions interviewed by FP6 thematic area FP6 Area Priority 1 Life Sciences Priority 2 Information Society Technologies Priority 3 NMP (Nanotechnology…) Priority 5 Food Quality and Safety Priority 6 Sustainable Development Priority 7 Citizens and Governance Specific international co-operation activities INCO Specific research activities for SMEs Human Resources and Mobility Research and Innovation Research infrastructures Science and Society Nuclear Energy Horizontal Strategy Committee Total
399
N. interviews 2 1 2 7 9 1 1 1 2 2 1 2 1 2 34
We discussed the idea of added value with the UK delegates to the PMCs of FP6, and in most cases, with both the departmental and scientific delegates. Overall, there was broad agreement across all interviewees, and for all parts of FP6, that Framework does offer added value.4 Table 8 summarises the main points of added value reported by UK PMC members and UK programme participants. Table 8 - European added value (EAV) of Framework Augmentation of national funds • Framework expands the funds available to national researchers over and above that which is available to them through national research funds alone • Intellectual gearing • Framework provides UK participants with access to foreign researchers and research outputs in a way that national funds cannot Other types of added value • Scale - pooling of resources as a means by which to increase investment in common European issues, from food safety to climate change • Scope - pooling of competence as a means by which to increase the likelihood of a breakthrough in a given area from the economic manufacture of large structural composites to the sequencing of plant genomes Strong added value in terms of the knowledge stock (science) • Complex issues resolved more quickly and more thoroughly as a result of larger projects and portfolios multiple projects across successive Frameworks • Status of knowledge accelerated through diversity and competition among national research traditions Increasing added value in terms of support to EU policy • EU regulates a growing number issues such as environmental protection or food safety • Framework has made substantive additional investments in science, in areas such as climate change and infectious diseases • Framework contributes to a more coherent EU view on risks and mitigation strategies • However, arms-length involvement of policy makers limits real impact Added value to EU businesses focused on key sectors • Builds in-house competence, tools and de facto standards • Strengthens international relationships
The research councils tend to see EAV most positively, with a strong endorsement of Framework in principle and in practice. They talk in terms of the additional scale and scope of the research, which can make for better science and faster progress. 4
A description of FP6 priorities and programmes can be found online at http://europa.eu.int/comm/research/fp6/index_en.html.
400
This is particularly evident in fields such as environment or medicine, where science is seen as a global issue. PMCs have stressed how, in these research fields there are growing number of research topics where an international approach has the ability to achieve results where a national approach may not, because of high costs of infrastructures and because of the general interest of issues studied. The research councils were less emphatic about the value of, and need for, an international programme, when it came to the subject of access to particular non-national networks and areas of competence. UK scientists tend to be well networked internationally, and in many cases those links are being built and maintained naturally as a part of activities financed by national programmes. Government officials with an industry sponsorship remit tended to be more ‘pragmatic’ and principally viewed Framework as a source of money for applied research in an area of interest to them, which they see as a worthwhile expansion of funds available nationally and in some cases amounts to a fund for which there is no counterpart nationally. Government officials with a wider policy remit (i.e. not science or industry) see the FP as ideally placed to research questions to underpin European policy, in a manner that does add value and is appropriate in light of the influence of Brussels in the legislative process. People were attracted by the ‘gearing’ one gets with a European programme, which may be 10 times the scale of the equivalent national programme. With this kind of scale, policy makers assume that it ought to be possible to launch five projects not one, and that this parallel working should accelerate progress in terms of scientific breakthroughs in (i) understanding of risks, (ii) development of models and forecasts and (iii) development of new methodologies for diagnosis, mitigation or treatment. There was little feedback on the question of scope, in terms of the extent to which an international programme permits a community to gain access to competence not otherwise available nationally and critical to the success of a project, which we read as a neutral rather than negative response. Early interviews with participants suggest that this may be a rather more important dimension of EAV from the point of view of individual projects, than this view would lead one to conclude. There was a general view that the science and engineering base in the larger member states needs to be pretty good (approaching world class) in order to get into project proposals, and to win through to an award. So, there are limits to the opportunities for capability building in any fundamental sense. The UK representative of the Research Infrastructures specific actions sees a clear EAV in this line inasmuch as it provides a valuable additional revenue stream to co-finance design studies for new facilities in important 401
new areas, where perhaps member states are less likely to be willing or able to move forward in a unilateral sense. The Science and Society Action line generated a mixed response, with respondents arguing that there was a clear EAV from a programme targeted on questions of ethics, scientific advice and governance. These are the conditions within which all policies must be framed, and as such it makes good sense to pool resources and to push for a European position on these new and challenging issues. However, respondents from OST, the policy ministries and the research councils considered this to be an area of such importance that there was a case (to consider at least) for a concerted national effort to complement and feed into the work of the Framework. In its current guise at least, Framework should not be seen as an alternative to a UK research programme(s) developing understanding in a wide range of topics. Indeed, the interplay between national programmes and the Framework Programme was discussed along a number of dimensions, several of which were rehearsed as part of the arguments explaining the EAV of Framework. In the first instance, the research councils and policy makers tend to see a difference in terms of scale and scope of the funding. The Framework Programme is generally viewed as a means by which to tackle larger, more complex and more fundamental scientific questions than do national programmes. The second dimension of the national/Framework divide relates to funding. In most of the thematic programmes, the view from the industry sponsorship units is very much about the additional money that may be available to their sector. While the FP may not be the ‘optimum’ instrument for business support, officials stated that they would continue to pursue opportunities in Europe. The third dimension relates to the synergy between national and international programmes, rather than differences or uniqueness of those instruments. In almost all cases, respondents argued that to be successful in framework, a member state must have a pre-existing research base with an internationally recognised body of work and people and that these will tend to be built through national programmes. The importance of existing capability is no less important on the industrial side, albeit this is rather less of a function of national programmes. 3.5.3 UK participants views on EAV Discussions on EAV related issues with participants supported the views of the UK officials. Participants’ primary motivations for becoming involved in Framework are to access the research funding it provides, which enables them to tackle problems of international relevance and to do so in partnership 402
with international partners who bring complementary knowledge, skills and perspectives to the table. Table 9 presents data on participants’ motives for electing to undertake their project in collaboration with overseas partners. Table 9 - Motives for undertaking the project in collaboration with international partners (n=1,255)
Motive To access research funding To tackle problems with a European or international dimension To access capabilities that do not exist in the UK To share the costs / risks associated with the project To access facilities that do not exist in the UK Other
Share of Share of participants participants ranking 1-3 ranking 1 86% 42% 79% 51% 45% 27% 10%
32% 11% 8% 4% 3%
Different types of organisations provided different responses. Large firms and SMEs were significantly more likely to be motivated by a desire to share the risks associated with the project and its exploitation, rather than accessing Commission funding. Exactly the reverse was true for universities, and other primarily research performing organisations. A majority of respondents (73%) stated that international collaboration in R&D is becoming more relevant to their organisation. Responses provided by large enterprises centred around three main interrelated issues. The most frequently mentioned concerned the globalisation and integration of markets. Large enterprises tend to have global or at the very least multinational operations, and in order to innovate and improve their competitiveness they collaborate closely with customers, suppliers and other units located overseas. The second main reason is that international collaboration offers access to resources not available locally. In some companies financial resources for R&D have been reduced significantly in recent years (especially for the more generic types of research), and collaboration with partners in other countries is attractive due to economies of scale and access to additional sources of funding. For some other companies, increases in international collaboration are driven by a need for new knowledge and competencies that are not available locally. The third most frequently mentioned reason concerned a requirement to keep pace with competitors. 403
SMEs expressed two main sets of reasons for the increasing relevance of international collaboration in R&D. First, international collaboration is very important, some say essential, if SMEs want to export and expand to international markets. For those already integrated into the international markets, customers and suppliers are from many countries and collaboration is critical in order to effectively manage those markets. Second, international collaboration provides access to complementary skills and capabilities that cannot easily be found in the UK. Some of these SMEs operate in markets that are heavily dependent on science and where the developments are becoming increasingly complex and interdisciplinary. For these, international collaboration in R&D is vital to ensure that they have the up-to-date knowledge that they need in order to remain competitive. The reasons provided by private research institutes mirror those given by the large enterprises. On the one hand, their client base is constantly expanding towards new regions, and on the other, important scientific developments are happening around the world. In order to remain competitive these institutes need to be part of this global effort. A further driver is a reduction in the scientific and technical resource available at the local level: a shrinking UK manufacturing base and decline in certain areas of British science were mentioned as reasons to look for R&D partners in other countries. In addition, several institutes mentioned the increasing complexity of new technologies and the existence of specialised knowledge in other countries. Universities and colleges offered many reasons for increasing international research collaboration, of which the four most common are: Research projects are increasingly multidisciplinary and occur on European and global scales. Similarly, the development of new technologies is increasingly accomplished at the trans-national level Multidisciplinary and globally-scoped research projects demand access to a broad range of knowledge and expertise, aspects of which may not be available at the local level. Collaboration in complementary subject areas is needed Some universities and colleges, especially the small ones, feel that UK funding is more difficult to secure than international funding Codes and standards are increasingly being developed on the European and International levels, and many technological and scientific developments are driving, and being driven by, these developments Lastly, the comments of public research institutes can be grouped into two main areas. First, since scientific questions are becoming more complex, pooling competencies from across many countries accelerates progress and 404
leads to stronger results. Second, national funding is shrinking in some areas, causing institutes to seek other funding sources including the EU. The principle of subsidiarity dictates that the Commission should not support activities that would be better conducted at a national level. Participants were therefore asked if a UK research programme could have supported their project. Most participants argued that their project would not have attracted national funding: The responses suggest that Framework satisfies the principle of subsidiarity; over three-quarters (77%) of participants stated that their project could not have been supported nationally. Figure 2 shows the reasons why projects could not have been supported nationally. Figure 2 - Reasons why projects could not have been supported nationally (n=939)
No support for vital international partners Addressed an EU rather than UK issue No funding for type of activity performed No funding in the same research area Too commercially-oriented Sought national funding but were refused 0%
10%
20% 30% 40% Percentage of respondents
50%
60%
The majority of UK participants report high levels of additionality for their projects. Our survey asked participants what would have happened to their project if it had not managed to secure Framework funding. Figure 3 shows that, in the views of the participants, the vast majority (70%) of Framework projects would not have gone ahead without Framework support. In these cases we can conclude that Framework is funding work that is wholly additional, at least from the UK partners’ point of view. This is a high figure and may reflect the composition of the UK participation: the majority of participants are research performers (public and private) pursuing income to sustain activity in relevant topic areas and not to secure funds in order to conduct a specific project that would be blocked otherwise.
405
Figure 3 - Participants’ views on additionality of FP funding
Would have done another project instea 7%
Would have proceeded with the project but wit alternative sources of funding 6%
Would have proceeded with the project but on modified basis 17%
Would not have gone ahead with the project all 70%
In a minority (7%) of cases, participants stated that they would not have gone ahead with the project if it had failed to secure Framework funding, but that they would have become involved in a different project instead. We can therefore conclude that Framework, in these cases, has displaced alternative projects that might otherwise have been carried out. There is no way to determine, however, whether these ‘other’ projects would have been more or less valuable than those that were undertaken within Framework. In a further 6% of cases the UK participant would have sought alternative sources of funding for the project, suggesting that in these cases the project was important enough to warrant continuing with even if Framework had not agreed to support it. The remainder (17%) of the participants indicated that they would still have proceeded with the project but with either fewer partners (64%), a reduced scope (90%) or a longer timeframe (59%). In these cases we would say that there is partial additionality, since the funding permits projects to proceed more quickly, or with more partners, or with a wider scope, or a combination of these. 4. Conclusions This paper illustrates the participation of UK organisations into recent framework programmes. We considered the extent of UK participation into FP4 and FP5 and provided an evaluation of the goal attainment in Framework projects. The purpose of our analysis has been to investigate the added value of research undertaken at the European level with the support of Framework funding.
406
We analysed EC data on participation in FPs, the responses from a survey targeting UK participants in FP4 and FP5 and the responses from policy makers directly involved in Framework policy. International R&D programmes offer unique potential as compared with (even large) national programmes. However, little evidence has been found that clearly demonstrates a general advantage in terms of Europe-wide research collaboration, beyond a restatement of statutory principles, in comparison with national programmes or other forms of international research cooperation. In principle, international – and European-level – collaborations may be expected to go beyond the limitations of national research programmes especially where a ‘critical mass’ of research performers – and users – is unlikely to be achieved nationally, or where a national focus would exclude ‘key players’ from research joint ventures. The empirical evidence reported here confirms the importance of international collaboration at a European scale. Overall, both policy makers and all categories of participants in FP4 and FP5 endorsed the need for a substantial research fund to support European industrial competitiveness and policy. Our survey confirmed that there is broad agreement regarding the fact that Framework does offer added value and that the additional scale and scope of FP research can accelerate scientific progress. FP participants also confirm the relevance of the international dimension of FP projects. Despite the primary relevance of accessing research funding in becoming involved in Frameworks, the majority of participants agrees on the fact that such funding enables them to tackle problems of international relevance in partnership with international partners who bring complementary knowledge, skills and perspectives, and who could have been involved if project were supported by national funding schemes. Similarly, confirming the key role of the European dimension of FP funding, our survey confirmed that without FP funding in the majority of cases projects would have not gone ahead. Overall, the case of the UK proves that Framework is effective in providing a European dimension to scientific research. Framework does have a strong rationale, in a general sense at least, and does provide added value on the input side for both UK researchers and policy makers. Indeed, participation in FPs would make little sense, unless set primarily to tackle questions that are uniquely European. However, the results of our study raise some questions about who is benefiting the most from framework research. Unlike in other European countries, at present UK participation in FPs is strongly biased toward HEIs and public research centres. Universities and research institutes score highest levels of benefits achieved and also seem to gain the most out of FP projects in terms of EAV. Research funded through framework suits particularly well this kind on institutions whilst leaves, for a number of reasons, partially 407
dissatisfied private industrial organisations, to the point that in the near future around half of them will be less likely to apply for framework funding. If the EC is wishing to invest on industry networking and if Frameworks can be seen as a viable instrument for stimulating firms’ collaboration in research, then this trend must be reverted and the needs of industrial organisations accommodated. In conclusion Framework seem to offer extensive additionality, supporting international research projects that otherwise would not be pursued. Framework funds research with a considerable EAV and offers participants substantial benefits in terms of knowledge generation. However, Framework evaluations tend to focus on actual participants in research projects and can offer little evidence of long-term socio-economic benefits deriving from framework research. In other words, we know that frameworks grants EAV for participant organisations; what we do not know is what impact this scheme has on the downstream users of the results produced by framework projects. Therefore, more efforts should be dedicated to investigating framework research from a user perspective and in the estimation of spillovers generated by framework projects. Tackling these aspects in future framework evaluations would greatly improve our understanding of the true added value of European research.
408
REFERENCES ARNOLD, E. – CLARK, J. – MUSCIO, A. (2005), “What the Evaluation Record tells us about EU Framework Programme Performance”, Science and Public Policy, forthcoming. BRANSCOMB, L.M. – KELLER, J.H. (1998), Investing in Innovation: Creating a Research and Innovation Policy that Works, Cambridge, MIT Press. COOMBS, R. – RICHARDS, A. – SAVIOTTI, P.P. – WALSH, V. (1996), Technological Collaboration: The dynamics of cooperation in industrial innovation, Edward Elgar, Cheltenham. DODGSON, M. (1993), Technological Collaboration in Industry: Strategy, Policy, and Internationalization in Innovation, Routledge, London. EC (1997), Five-Year Assessment of the European Community RTD Framework Programme, independent panel chaired by Viscount E. Davignon. EDQUIST, C. (1997), Systems of Innovation. Technologies, Institutions and Organizations, Pinter Publishers, London and Washington. FAHRENKROG, G. – POLT, W. – ROJO, J. – TÜBKE, A. – ZINÖCKER, K., (eds.) (2002), RTD Evaluation Toolbox: Assessing the Socio-Economic Impact of RTD Policies, STRATA Project HPV 1 CT 199900005, IPTS Technical Report Series, EUR 20382 EN, IPTS, Seville. GEORGHIOU, L. (2001), “Evolving frameworks for European collaboration in research and technology”, Research Policy 30, pp.891-903. GEORGHIOU, L. – RIGBY, J. – CAMERON, H. (2002), Assessing the Socio-economic Impacts of the Framework Programme, PREST, University of Manchester. GRAVERSEN, E.K – SIUNE, K. (2000), Danish Research Co-operation in EU: Extent, Return and Participation, An analysis of co-operation in the 4th EU Framework Programme, Danish Institute for Studies in Research and Research Policy, Århus, Report 2000/7. GUSMÃO, R. (1999), Developing and Using Indicators of Multilateral S&T Co-operation: the Example of European Research Programmes, OST, Paris. ISG, Institut für Sozialforschung und Gesellschaftspolitik (2001), Europäische Forschungsrahmenprogramme in Deutschland, ISG, Köln. KATZ J.S. – MARTIN B. (1997), “What is a Research Collaboration?”, Research Policy 26, pp.1-18. KUHLMANN, S. – EDLER, S.J. (2003), “Scenarios of technology and innovation policies in Europe: Investigating future governance”, Technological Forecasting and Social Change 70, pp.619-637. LUUKKONEN, T. (1998), “The difficulties in assessing the impact of EU framework programmes”, Research Policy 27, pp.599-610. 409
LUUKKONEN, T. (2000), “Additionality of EU Framework Programmes”, Research Policy 29, pp.711-724. LUUKKONEN, T. (2002), “Technology orientation in company participation in the EU Framework Programmes”, Research Policy 31, pp.437-455. MARSHALL, A. (1890), Principles of Economics, MacMillan, London. MIOTTI, L. – SACHWALD, F. (2003), “Co-operative R&D: Why and with Whom? An Integrated Framework of Analysis”, Research Policy 32, pp.1481-99. DUNNING, J. – NARULA, R. (1998), “Explaining International R&D Alliances and the Role of Governments”, International Business Review 7, pp.377-397. NIFU-STEP – TECHNOPOLIS LIMITED (2004), Evaluation of Norway’s Participation in the EU’s 5th Framework Programme, Oslo. PAVITT, K. (1998), “The inevitable limits of EU R&D funding”, Research policy 27(6), pp.559-568, special issue on EU research funding policy. PREST – AUEB – BETA – ISI - JOANNEUM RESEARCH - IE HAS, WISE GUYS, (2002), Assessing the Socio-Economic Impacts of the Framework Programme, University of Manchester. SCHIBANY, A. – JOERG, L. – GASSLER, H. – WARTA, K. – STURN, D, - POLT, W. – STREICHER, G. – LUUKKONEN, T. – ARNOLD, E. (2001), Evaluation of Austrian Participation in the 4th EU Framework Programme, (BMVIT, Vienna). SOETE, L. (2005), Activating Knowledge, Discussion paper prepared for the UK Presidency, United Nations University, Maastricht. SQW Ltd (1999), Impact of the EU Framework Programmes in the UK, a report for the DTI. TECHNOPOLIS LIMITED (2001), The 4th Framework Programme in Ireland, prepared for Forfas, Dublin. TECHNOPOLIS LIMITED (2004), The Impact of the EU Framework Programmes in the UK, prepared for DTI – OST. UOTILA, M. – KUTINLAHTI, P. – KUITINEN, S. – LOIKKANEN, T. (2004), Finnish Participation in the EU Fifth Framework Programme and Beyond, Finnish Secretariat for EU R&D, Helsinki. VON TUNZELMANN, N. – KRAEMER-MBULA, E. – MEYER M., NASSEHI S., PATEL P., RANGA, M. (2004), Study on the new instruments in FP6 - use and effect, SPRU - prepared for the ESRC, May. VON TUNZELMANN, N.– RANGA, M. – MARTIN, B. – GEUNA, A. (2003), The effects of size on research performance: a SPRU review, SPRU – prepared for the Office of Science & Technology, DTI, June. YELLOW WINDOW MANAGEMENT CONSULTANTS SA/NV TECHNOFU SA - WISE GUY LTD, (2000), Identifying the Constituent 410
Elements of the European Added Value of the EU RTD Programmes: conceptual analysis based on practical experience; Final Report for the DG XII.
411
