Evaluation of the Research Council of Norway

(1)

31 August 2012

Evaluation of the Research Council of Norway

Background Report No 1. - Production of Strategic Intelligence and Advice

Erik Arnold Bea Mahieu Malin Carlberg

(2)

Evaluation of the Research Council of Norway

Background Report No 1. Production of Strategic Intelligence and Advice

Erik Arnold, Bea Mahieu. Malin Carlberg technopolis |group|, August 2012

(3)

Table of Figures

Figure 1 RCN’s strategic intelligence production and the advisory function ... 6

Figure 2 Research and innovation system and the reach of public policy...7

Figure 3 Expenses for knowledge creation – Innovation Division and Science Division 2009 ...10

Figure 4 Type of committees or boards (2003-2010) (Percentages)... 15

Figure 5 Regional participation in RCN committees, 2003-2011...16

Figure 6 Some Key Policy Documents in RCN and the Ministries ... 38

Figure 7 Priorities of the 2004/5 White Paper... 40

Figure 8 Sources of income in the institute sectors, 2009...55

Figure 9 Institutional funding channelled through RCN, 2000-2010 ... 59

List of tables

Table 1 Stakeholder involvement in RCN management committees, per division ...16

Table 2 Meeting place function: views on results from participation in RCN ‘strategy meetings’ ... 17

Table 3 Evaluations undertaken by RCN 2001-2011... 20

Table 4 Evaluations of Research Institutes at the Organisational Level, 2006-2010 .. 25

Table 5 Discipline evaluation including institutes, with conclusions at the institutional level...27

Table 6 Discipline evaluations including departments or divisions in research institutes, with conclusions at department/division level ... 28

Table 7 How RCN strategic advice was reflected in the 2008/9 White Paper ...41

Table 8 Categorisation of the research institutes ...57

(5)

Summary

This report discusses RCN’s functions in collecting and using strategic intelligence and in giving advice. Other background reports touch on these matters, too, and the evidence from them is used in conjunction with that from this report in reaching the overall conclusions and recommendations in the synthesis report.

We cannot evaluate RCN without reference to the overall governance context. Norway lacks an effective national research and innovation council in the Finnish style – a style that is increasingly imitated in various ways around the world. While KD has lead responsibility for research coordination across the government and for RCN, its ability to coordinate is constrained by the sector principle and the lack of a higher- level ‘referee’ such as a research and innovation council. This in turn regulates the effectiveness of RCN as an advice-giver.

Strategic intelligence

We distinguish between strategic intelligence and advice. Strategic intelligence – in the sense of the knowledge needed to make strategy but also the deliberate use of evaluation, foresight and technology assessment in policy formulation and implementation – is a characteristic of research and innovation systems that needs to be decentralised, to enable components of the system to work well. It is not enough that one central actor knows everything – knowledge must be developed and particularly shared across the system of actors involved.

Today, in partnership with others (eg SSB, NIFU), RCN generates a lot of ‘systems health’ indicators. These are now complemented by KD’s ‘research barometer’. It continues to be very open in the extent of its consultations with stakeholders. A number of research roadmaps for discipline development have been developed. These are valuable and play a role in guiding events among the research performers.

RCN undertook a number of research foresights from 2004 onwards. The use of foresight studies has since declined to a low level. RCN continues the practice of broad stakeholder consultations for the development of new programmes, but there is an ongoing lack of proper foresight exercises. Stakeholders regard RCN as an important arena for counselling and dialogue on research and innovation policies. However, meeting place participants tend to see participation as an opportunity to learn and to network rather than as a chance to influence RCN policy or practice. This is true both at the level of stakeholder meetings and in RCN’s Boards.

RCN’s expenditure and activity in evaluation are modest – this is an under-used tool.

Field evaluation is an area where RCN does well – it has inherited and improved the NAVF tradition. This is normally followed up, with the evaluated fields being supported to develop a research road map. Where relevant, this can have an effect on future RCN programmes. However, we can see little evidence of work that problematises new fields or that tackles the problems of interdisciplinarity.

Evaluation is otherwise poorly institutionalised. The evaluation strategy dates from 1997 and has never been implemented. Evaluation is not embedded in the programme or policy cycle. There is a lack of meso-level studies. There is little interest in impacts.

However, there is growing use of evaluation in relation to larger programmes.

Dropping the evaluations of the institutes means that there is now no institutional perspective on their performance and that their chief source of feedback is instructions from their parent ministries and signals from markets.

(6)

Advice to government

RCN plays an important role as a co-operator in the production of strategic intelligence and policy. Since it does not work at the political level of the ministries and since – despite being uniquely well positioned to generate intelligence and advice – it does not have a monopoly of knowledge, it would be odd if it were otherwise.

The annual budget proposals are argued to be a key source of advice to government.

They result from 16 intimate, parallel and increasingly detailed dialogues between RCN and the ministries. Ministries’ needs of and attitudes to RCN differ widely. It is very complex for RCN to handle this diversity – it therefore acts case by case.

However, RCN has succeeded in signing up more and more ministries to a declining number of common programmes, so it clearly is able to set or exploit cross-ministry agendas and find synergies in R&D funding.

At the level of national policy, RCN is a big and active participant in a debate that involves many other actors in addition. However, RCN advice on national policy tends to reflect the fragmented nature of its dialogue with the ministries. It sometimes produces proposals orthogonal to the contents of those discussions but there is not a clear whole-system vision from which RCN generates such advice. We argue that this results at least in part from decentralising the production of strategic intelligence within RCN to the divisions.

RCN’s ability to ‘sell’ individual programmes to multiple ministries is an important (if labour-intensive) form of coordination. The ministries themselves show signs of coordinating specific strategies, such as those for generic technologies. In these cases, RCN aims to contribute strategic intelligence and secretariat support to the process of developing strategy. RCN coordination from ‘below’ to a degree complements inter- ministry coordination from ‘above. (In other words, the system is itself evolving to cope with the coordination deficit at the highest level.)

At the government level, the sector principle is very valuable. While KD has responsibility for coordinating research policy, in practice it has limited authority. A consequence is that there is only in a limited sense a national strategy – that is the strategy that KD can negotiate with the other ministries during the White Paper and budget processes. There is no higher-level mechanism for creating a view that goes beyond the individual ministry views or the sum of ministry views where they choose to develop national strategies together, eg in bio- and nano-technology. This is increasingly problematic as the locus of research policymaking in Europe shifts towards Brussels

Advice to the research performers

RCN and others have identified needs for structural change in the research system, notably to tackle fragmentation, lack of mobility and the need for greater internationalisation. Government policy has been to make the research performing organisations more autonomous. Hence, the ‘advice;’ that RCN can give them has to be given at least in part through providing incentives. The three Centres programmes (SFF, SFI and FME) provide clear signals about building critical mass, training researchers and better international exposure. Other measures such as the research infrastructure plans developed in 2006 and subsequently similarly promote de- fragmentation and a better division of labour in the research system. More generally, RCN influences research performers through its thematic and non-thematic programmes.

RCN’s strategic responsibility for the research institutes

RCN has always had ‘strategic responsibility’ for the research institutes – a responsibility it has been difficult to fulfil, given its lack of authority over them and lack of control over budget. RCN’s main steering instrument has been its

(7)

programmes. In line with the government’s policy to make research-performing institutions more autonomous, RCN helped develop a new performance-based research funding (PBRF) system for reallocating parts of the core funding among the institutes. It also revised its programme for providing ‘strategic’ funding to institutes, to help them develop capacity. The PBRF has affected the behaviour of many of the institutes. Not all the ministries have been prepared to transfer core funding into the PBRF-based part of their funding arena and only one area has so far implemented the new strategic programme. There has been little restructuring in the institute sector.

Internationally, the use of a PBRF in the institute sector is unusual but not unique. In developed countries, it is similarly unusual for all the institutes to have a single owner, but they are often grouped under umbrella ‘owners’ (like the Fraunhofer Society) in order to give common management to common categories of institute. But evaluation and funding tend to be done at the same level, so that evaluations have consequences.

And where there is a need to steer the portfolio of institutes, it is done by active management rather than using indirect incentives such as PBRFs.

Conclusions

RCN has a substantial list of achievements to its credit. In many cases these cannot solely be attributed to RCN because they are produced in partnership with others. The ones we mention here are nonetheless ones where RCN has at least played an important role – and our list is not exhaustive.

• RCN produces or co-produces a very large volume of strategic intelligence at the level of indicators and surveys. These range from the Indicators Report to detailed monitoring of the research institutes. They are of general interest for making and implementing policy

• Strategic intelligence and policy are developed in the context of large-scale stakeholder consultation. This is difficult to benchmark but is certainly towards the most consultative end of the spectrum of policy development internationally

• Field evaluations are regularly conducted and provide information that is valuable to participants and their organisations as much as it is to RCN itself. These have consequences for participants’ strategies and for RCN programmes

• Evaluation is to a growing extent informing RCN programming beyond disciplinary research (to which the field evaluations are primarily relevant)

• RCN plays a significant role in helping sixteen ministries plan a large and growing part of their research expenditure. The budget is a key process for doing this.

While there are two parts to this discussion – one on the next year and one on the following year – a longer-term element might be beneficial

• RCN is an active and well-informed partner for ministries responsible for writing White Papers. The main interaction is with KD for the research White Paper, but there are also others

• RCN supports the coordination of sectoral research needs by developing and implementing research programmes of interest to multiple ministries. In this way, a declining number of programmes is satisfying the needs of a growing number of ministries (in the sense that the mean number of ministries per programme is increasing)

• Ministries are, singly and in groups, preparing thematic national strategies on research. RCN is increasingly providing coordination by supporting these with strategic intelligence and by providing or hosting secretariats

• These two coordination mechanisms appear to be evolutionary adaptations to the lack of an overall ‘referee’ in the policy system

• RCN is playing a significant role in the development and deployment of programmes that tackle structural deficits in the research system, including the Centres programmes (SFF, SFI and FEM), research infrastructure and the regional research funds. These systemic interventions tend to lie outside the interest of individual sector ministries and have been tackled using money from the Research and Innovation Fund. This underscores the importance of RCN as a

(8)

change agent and the need for ‘strategic’ resources to be available to counterbalance the tendency of sector-driven funding to cause lock-in

• RCN has made a major contribution towards strengthening the institute sector by designing and implementing the new performance-based funding system, even if that system has by no means been fully rolled out at this stage

Issues and problems raised in this report include the following.

• RCN made use of foresight for a short period but seems since largely to have dropped it. Foresight is a useful component of strategic intelligence because it helps you move away from consensus to explore disruptive possibilities and counteract the tendency of research agendas and programmes to lock in to existing ideas and trajectories

• Equally, we were not able to identify much strategic intelligence about interdisciplinarity or new and disruptive directions in research

• Evaluation is not properly embedded in the programming cycle at RCN. While we are cautious of the idea that everything has always to be evaluated, formally deciding whether to evaluate before, during or after a programme and in relevant cases doing such evaluations ought to improve the quality and efficiency of intervention

• Nor does evaluation adequately tackle impacts. As a result, RCN lacks evidence for accountability and to demonstrate the value of what it does

• The European and global context means it is increasingly important to have a clear national strategy in relation to quality, thematic focus, internationalisation, etc.

Without this a small country easily becomes irrelevant in the international research system and resources are wasted on sub-critical and fragmented efforts.

Given the lack of a ‘referee’ in the system, such a strategy is hard to make truly national in Norway

• Advice giving to government appears overly embedded in RCN’s interactions with the ministries. RCN needs the capacity to develop strategy and advice that is not captive to the ministry agendas and that therefore has greater potential to induce disruptive change

• The availability of strategic resources in the form of the Research and Innovation Fund has been key to RCN’s ability from time to time to act as a change agent.

Replacing the Fund with a line in the KD budget exposes it to the short-term budgeting process and therefore political risk

• The reform of the research institute system is unfinished business. Neither component of the new funding system is fully implemented. The incentives for restructuring the system remain rather weak and the interest of a number of ministries in addressing institute policy seems limited. The end of evaluation means there is no rounded view of the individual institutes as organisations.

International experience with performance-based funding systems suggests that strongly formula-based steering leads to perverse behaviour and lock-ins. We therefore would prefer to see a mix of measurement and judgement by one or more competent owners of clusters of institutes rather than treating the institute system as a quasi-market. But whichever view one takes, the institute system currently hangs between an evaluation-based system that had no ‘teeth’ and a performance-based system that is only partly implemented. This is clearly not satisfactory.

(9)

1. Introduction

The Research Council of Norway has three main functions: it acts as an executive agency, managing public funding of research and innovation; it is an advisory body, expected to provide input to government authorities as a basis for the formulation of research policy; and it is to provide an open arena for counselling and dialogue. RCN typically describes these tasks as its funding, advising and meeting places functions.

RCN’s original statutes say that it shall “provide advice as a basis for the development of the government’s general research policy.”¹ The 2004 and 2011 statutes² both say

“The Research Council shall serve as an advisory body to the government authorities on matters concerning research policy.” Both versions underline that “The Research Council’s Executive Board [HS] shall follow up the research policy guidelines drawn up by the Government and the parliament, and shall in its advisory capacity to government provide input for future research policy.” HS is to oversee the creation and implementation of RCN’s own strategy.

The difference between the two formulations of RCN’s advisory role is subtle but important. In our reading of history, RCN was originally created as a way to reduce fragmentation in research and innovation funding, enabling holistic research and innovation policies to be developed and implemented. Its role as advisor to the government on research and innovation policy replaced the idea of a separate high- level committee t advise the government – following a long history of such committees proving ineffectoive. The reformulation of the relevant statute reflects (as we understand it) a perception that it was difficult for RCN to be an independent policy advisor at the same time as being a major policy actor; and that the government recognised the importance of obtaining research and innovation policy advice from multiple sources.

In practice, RCN expects and is expected to ensure the provision of several forms of information and advice.

• Information (strategic intelligence) as a basis for analysing the health of the Norwegian research and innovation system and in connection with the development and deployment of RCN programmes

• Advice

− To government overall about research and innovation policy as well as advice about how to implement national priorities

− Thematic advice to individual ministries about their research budgets and priorities– advice that appears to be more appreciated by some ministries than by others³

− On specific needs in the research and innovation system, such as the development of research infrastructure or of the university colleges and structural instruments such as centres of excellence

1 St. meld. Nr. 43, Et godt råd for forskning. Om endringer i forskningsrådsstrukturen, 1991-92

2 The Research Council of Norway, Statues, Oslo, 2004; The Research Council of Norway, Statutes, Oslo, 2011

3 RCN, Kunnskapsbaserte råd, virkemidler og møteplasser: Policy for Forskningsrådets arbeid med kunnskapsgrunnlaget – med focus på det tverrgående kunnskapsgrunnlaget, Oslo: RCN, 2011

(10)

• A combination of incentives and advice to research performer in the form of guidance about discipline development from scientific evaluations and action plans that respond to development needs

• And by taking what the statutes call “strategic responsibility for the research institute sector”

In Figure 1, we depict the various tools and instruments that RCN uses to create its strategic intelligence (the ‘knowledge base’) and for giving advice. In the wider Nordic research and innovation programming tradition, to which RCN historically belongs, strategic intelligence derives from at least two sources. One is analysis; the second is

‘presence’, in the sense of active engagement with researchers, the users of research and other stakeholders ‘on the ground’ both via the ‘meeting places’ and through routine interaction.

Figure 1 RCN’s strategic intelligence production and the advisory function

This report provides the primary analysis of RCN’s advisory function and the degree to which it is supported by the meeting place function. It is structured in four parts.

• First, we discuss the idea of ‘strategic intelligence’, the role we expect it to play in relation to the steering, strategy and operation of RCN and the role of evaluation as part f strategic intelligence

• Second, we analyse how RCN uses strategic intelligence to perform its advice- giving role in relation to government

• Third, we look at RCN’ use of incentives to restructure the research system as forms of ‘advice’

• Fourth, we discuss RCN’s role in taking ‘strategic responsibility’ for the research institutes

• Finally, we draw conclusions and make recommendations

CounsellingUseCreation

RCN aggregation & adoption of the knowledge Portfolio

analysis Programme

monitoring Evaluation Stakeholder

consultations Context analysis

Research roadmaps

Budget

negotiations Structuring the

system Thematic priorities

& strategies

Government

authorities Research

communities

Meeting places Conferences, meetings with stakeholders,

etc.

RCN steering committees

National committees International

committees

(11)

2. Strategic Intelligence

In this section we begin by discussing the idea of ‘strategic intelligence’. What is it?

How does RCN make it? We consider its production through externally commissioned studies, stakeholder consultation (including foresight) and ‘meeting places’.

Evaluation is an important source of strategic intelligence, so we analyse the way RCN does and uses it.

2.1 What is strategic intelligence?

The idea of ‘strategic intelligence’ has become important in the way we think about research, innovation and the institutions in which these happen in last 25 years or so, since the idea of ‘national innovation systems’ took hold. The old ‘linear model’ saw the link from research to generating new knowledge, innovation and wealth-creation as automatic and was completely unconcerned with how the links between successive stages in this innovation chain were made – or, indeed, what happened at each stage.

The innovation systems view takes account of these things, of bounded rationality and of institutions. National innovation systems are nationally specific not only because of differences in factor endowments, geography and culture but also because they co- evolve with national systems of governance. That is why government and its influence is one of the major components in the way we typically sketch an innovation system (Figure 2). This influence needs to be exerted on the basis of good information and advice.

Figure 2 Research and innovation system and the reach of public

Source: Stefan Kuhlmann and Erik Arnold, RCN In the Norwegian Research and Innovation System, Background Report No 12 in the Evaluation of the Research Council of Norway, Karlsruhe: Fraunhofer-ISI, 2001

If strategy is (after Clausewitz) “the employment of battles to gain the end of war” then

‘strategic intelligence’ is, properly, the information and knowledge needed to determine strategy. The term entered the research policy vocabulary in the late 1990s, as a result of the ASTPP project, an EU-funded cooperation among R&D evaluators, foresight and technology assessment specialists, which defined it as

(12)

… a set of – often distributed – sources of information and explorative as well as analytical (theoretical, heuristic, methodological) tools employed to produce useful insight in the actual or potential costs and effects of public or private policy and management. Over the last two decades, considerable efforts have been made to improve the design and conduct of effective research, technology and innovation policies. In particular, formalised methodologies, based on the arsenal of social and economic sciences have been introduced and developed which attempt to analyse past behaviour (evaluation), review technological options for the future (foresight), and assess the implications of adopting particular options (technology assessment).⁴

In the innovation systems view, governance is not a simple matter of top-down

‘steering’, in which an all-knowing principal sets agents to work to achieve goals that can be set from the top alone, but involves competition, consensus-building, networking and negotiating decisions in arenas in which multiple actors are involved⁵. According to Kuhlmann, the research literature indicates that the actual practice of governance within the research and innovation system sketched in Figure 2 is characterised by

• A high degree departmentalisation, sectoralisation of the political administration, and low inter-departmental exchange and cooperation

• Heterogeneous, non-inter-linked arenas: often corporatist negotiation deadlocks (e.g. health innovation related policy in Germany)

• Failing attempts at restructuring responsibilities in government because of institutional inertia

• Dominance of the ‘linear model’ of innovation in policy approaches (and of related economists as consultants) in many national authorities (e.g. ministries)

Tackling these problems – making sure that the different parts of the system can cooperate, find intelligent divisions of labour and where necessary work towards common goals – requires that strategic intelligence be distributed about the system.

Analysts do not have a monopoly of knowledge; inputs from others, such as those who work with stakeholders or in the laboratories, are also needed. The system of distributed intelligence therefore needs to: be networked; involve active actors or

‘nodes’ in the different organisations involved; be transparent so that as many parts of the innovation system as possible can share intelligence; publicly supported, so that there are resources available to provide data and analysis; and quality-assured through the participation of multiple providers of intelligence and regular efforts to keep the knowledge involved up to date.⁶

In this version of strategic intelligence, then, the producers of strategic intelligence are better seen as co-operators than as leaders. The ability to produce strategic intelligence at multiple levels of the research and innovation system needs in turn to be matched by absorptive capacity: the ability to identify and exploit it.

4 Stefan Kuhlmann, Paries Boekholt, Luke Georghiou, Ken Guy, Jen-Alain Héraud, Philippe Laredo, Tarmo Lemola, Denis Loveridge, Terttu Luukkonen, Wolfgang Polt, Arie Rip, Luis Sanz-Menendez and Ruud Smits, Improving Distributed Intelligence in Complex Innovation Systems, Final report of the Advanced Science and Technology Planning Network (ASTPP), TSER Contract No SOE1-CT96-1013, Karlsruhe:

Fraunhofer-ISI, 1999

5 Renate Mayntz and Fritz W Scharpf, ‘Der Ansatz der akteurzentrierten Institutionalismus’ in (same authors) Gesellschaftliche Selbstregelung und politische Steuerung, Frankfurt: Campus, 1995

6 Stefan Kuhlmann, Governance and Intelligence in Research and Innovation Systems, address delivered upon the acceptance of the office of a Fraunhofer-ISI Professor of Innovation Policy Analysis at Utrecht University on 7 October 2002, Universiteit Utrecht, 2002

(13)

2.2 How does RCN produce strategic intelligence?

RCN generates a lot of formal strategic intelligence by commissioning studies from external service providers (Section 2.2.1). RCN documents and the research budget data tell us that RCN builds upon expertise in other (public) organisations for the collection of statistical data and context information. The three other main activities are stakeholder consultations (Section 2.2.2), ‘meeting places’ (Section 2.2.3) and evaluation (Section 2.3).

2.2.1 Externally commissioned studies

One of the most conspicuous pieces of strategic intelligence is the annual Indicators Report. Indiktarrapporten⁷ is a report published annually by RCN, NIFU and Statistics Norway, which describes and documents the Norwegian research and innovation system. It is produced using Norwegian R&D and innovation surveys, statistics from Statistics Norway and other relevant studies. The report produces and presents key indicators for Norwegian R&D&I with the purpose of presenting an overall view of Norwegian activity in R&D, higher education, science and technology. An English summary is also published to accompany the more comprehensive Norwegian version in order to reach an international audience.

The latest version stems from 2011. It compares data between 2003-2009 and encompasses the following chapters

• Norwegian R&D and innovation in an international context – investment, human resources, and results

• The national R&D and innovation system – human resources, total resources, R&D financing per sector, results of Norwegian R&D&I

• Regional comparisons of R&D and innovation – human resources, business survival rates and investment per county (fylke), and regional indicators.

Other studies provided by NIFU in the last 3 years were also predominantly on innovation and R&D statistics and indicators. Topics included bibliometric data analysis, Biotechnology R&D, Marine R&D, polar research, policies for the ‘knowledge society’ and barriers to commercialisation. The state statistical bureau (SSB) provided inputs such as a study on the regionalisation of industrial R&D and innovation, while Møreforsking undertakes annual monitoring of the effects of user-directed research projects. Other service providers include the Norwegian Business School (BI), SINTEF Technology & Society and the TIK. Group at the University of Oslo.

In 2009, studies commissioned by the Innovation division were predominantly focused on topics related to regional innovation (close to 60%), while more general studies on innovation and market studies accounted for 10% each. The Science Division invested especially in studies on the effects of research, including a study on R&D, Industry Dynamics and Public Policy (about 50%) and in scientific discipline evaluations (about 40%).

7 www.forskningsradet.no/servlet/Satellite?c=Page&pagename=indikatorrapporten%2FHovedsidemal&

cid=1224698172624

(14)

Figure 3 Expenses for knowledge creation – Innovation Division and Science Division 2009

Source: RCN, 2011 – Technopolis analysis

Most of the studies commissioned are ad hoc, and in its 2011 report on the knowledge base⁸, RCN reports on the internal reflection that “identification of conservative traits or other dysfunctions in the research system is very demanding, and must in some sense be regarded as a research task in itself”. RCN thus feels the need for both ad-hoc studies and longer-term research (e.g. through projects such as the newly launched FORFI project) that would provide more continuous but also more detailed understanding. Other agencies and some ministry personnel made the same point at interview – especially in relation to improving understanding of the innovation system as a whole rather than science.

In the 2001 RCN Evaluation, we estimated that in the years 1995-1999, the level of spending for evaluation represented no more than a maximum of 0.33% of the RCN research budget. Data related to 2003-2010 show that little has changed. As in the earlier period, RCN spends 1% of its budget on external services. On average approximately 30% of these external service costs are for studies building up strategic intelligence, while external evaluations accounted for about 20%. The 2011 RCN document on policy for work on the knowledge base⁹ shows that some projects that were intended to provide strategic intelligence were launched within the ‘normal’ R&D portfolio. These costs are additional to the ‘official’ costs indicated above.

In its 2011 document “Knowledge-based advice, tools and meeting places - Policy for RCN’s work in the knowledge base”¹⁰, RCN considered that in particular the knowledge base for R&D policy and instrument development needed improvement by means of more strategic intelligence studies, an improved communication with the stakeholders and cross-divisional knowledge sharing. The possibility was put forward to finance these activities through a small taxation of the programmes (as is current practice within the Innovation Division).

2.2.2 Stakeholder Consultations

In the mid 2000s, RCN launched a set of stakeholder consultation initiatives for the development of its programmes and strategies. These included set-up of scientific discipline committees that were to draw research road maps to follow up discipline evaluations, foresight exercises involving larger groups of stakeholders and wider stakeholder consultations. These initiatives were concentrated around the time when

8 RCN, Kunnskapsbaserte råd, virkemidler og møteplasser: Policy for Forskningsrådets arbeid med kunnskapsgrunnlaget – med focus på det tverrgående kunnskapsgrunnlaget, Oslo: RCN, 2011

Studies on regional innovation Other studies 58%

on innovation 12%

Studies in support for governance 10%

BIA Evaluation 10%

Market studies 10%

Expenses for knowledge creation - Innovation Division, 2009

base: 12.7 m NOK

Research on the effects of research 52%

Scientific discipline evaluations

37%

Evaluations of instruments and

institutions 4%

Studies on the research context

4%

Studies on tools for gender equality in research

3%

Expenses for knowledge creation - the Science Division, 2009 base: 16 m NOK

(15)

new large-scale programmes were being launched by RCN. Now that several of these are coming to an end, RCN is preparing new initiatives to support the design of the follow-up programmes.

2.2.2.1 Research roadmaps for discipline development

The studies for the development of research roadmaps (the ‘national plans’ – fagplaner) most often constituted a follow-up of scientific discipline evaluations.

These roadmaps included recommendations for systemic interventions and programmes to be launched or re-focused. Examples are:

• Strategic development plan for engineering science (2006)

• Strategic development plan for information and communication technology (ICT) (2004)

• Strategic development plan for mathematics (2004)

• Strategic development plan for political science (2004)

• Strategic development plan for bioscience (2003)

An example of such a strategic plan is basic engineering science. In 2004, RCN conducted a thorough academic evaluation of the major institutions in Norway offering education in basic engineering science at the master level: the Norwegian University of Science and Technology (NTNU), the Norwegian University of Life Sciences, the University of Stavanger and the Narvik University College. The international evaluation team, composed of 23 professors from leading universities in Europe and North America, considered all relevant professional groups at the four institutions. One of the main conclusions was that Norway should expand its critical mass in basic engineering science research. To follow up on this evaluation, the Research Council appointed a broad-based curriculum committee headed by the Rector of the NTNU, Torbjørn Digernes.

The purpose of the exercise was to establish a basis for the strengthening of basic engineering science research in Norway. An important part of the work of the committee was to describe the status and research challenges in eight thematic areas – petroleum technology, energy and environment, sustainable infrastructure, marine and maritime activities, materials, production, seafood’s value chain, process manufacturing [processindustri], and – overall – systems knowledge.

Based on the challenges and the specific research tasks within the different thematic priority areas, the committee defined a number of fundamental engineering science research topics that required research. The committee concluded its report, delivered in 2006, with a number of recommendations to the Norwegian Research Council, the ministries, as well as to the research institutions themselves.

In summary, the HE institutions were advised to create a Committee for Engineering [Fagråd for ingeniørvitenskap], which should be the main instrument with which to implement a national strategy. HEIs should also establish collaboration within and between the institutions in the prioritised thematic areas, with each institute developing a strategy for its themes.

The publication of research results in international journals should be encouraged and research groups balanced so that supported projects cover a range of engineering disciplines as well as basic and applied research. The goal should be to strive towards high quality research and to avoid fragmentation of knowledge production. The institutions should also establish a framework for business innovation and coordinate measures to increase the stimulation of innovation.

Recommendations aimed at the Research Council were to

• Develop and implement a national plan to strengthen basic engineering research

(16)

• In collaboration with the Norwegian Association of Higher Education Institutions (UHR), establish a Committee for Engineering to lead the implementation of the national plan

• Anchor the national plan in the thematic priority areas identified by the curriculum committee and implemented through a collaboration between RCN and the UHR

• Ensure that the national plan contributes to and develop areas of importance in Norway, and ensure that engineering becomes a ‘bridge from science to innovation’. The plan should also address recruitment

• Ensure budgets for basic science initiatives should be ring fenced, through for example the instrument Institution-based strategic projects [Institusjonsforankrede strategiske prosjekt] and independent of applied research budgets

• Ensure its Science Division should increase its support for basic engineering research

• Create ‘competition arenas’ that can assess the quality of Norwegian engineering research and which should be led by the Committee for Engineering and ensure resources are used effectively

• Continue and further strengthen initiatives such as the SSF and SFI, as these are good examples of activities to strengthen basic engineering research

• Support HEIs’ laboratories and high quality research equipment/ infrastructure and ensure HEI have access to funds for operational costs.

Third, recommendations aimed at the ministries focused on the steering of funds. The sector ministries should ensure that their allocations cover long-term and basic engineering research, and earmark funding for restricted engineering areas that play an important role in the national context, as well as develop a plan for upgrading scientific equipment.

The ministries should continue to support funding to strengthen Norwegian researchers’ publications in international journals and attendance at international conferences. They should specifically support publications produced by Norwegian researchers in collaboration with international researchers. The Committee report also called for the current bias in the reward system for PhD and postdocs should be adjusted, and for support the university colleges in their work to build strong research within niche areas. Furthermore, the departments should grant funding to the Committee for Engineering so it can engage in strategic initiatives such as the Framework Programme and other international schemes.

Follow-up of the work of RCN’s curriculum committee can be traced via RCN’s annual reports in subsequent years.

In 2006 RCN announced it was awarding NOK42m (with more than two-thirds awarded to NTNU) over a three-year period through the Institution-based Strategic Project instrument as a follow-up of the work of the Curriculum Committee. The funding was announced to the universities that were included in the evaluation and RCN monitored the use of the investment. The funds aimed to allow the recipient institutions to take on the recommendations made by the Committee, in particular doctoral training and a renewed focus on long-term basic research. Updates on the use of the funds were subsequently published in RCN’s annual reports.

The same year (2006) RCN also prepared a draft national strategy for the reinforcement of basic research, in close consultation with UHR and KD. This was delivered to KD in October 2006. It identified recruitment, the upgrading of equipment and infrastructure, professional development and internationalisation as strategic priorities.¹¹¹²¹³

11 Årsrapport 2006 – Del III, p.146

(17)

2.2.2.2 Foresight exercises

Following recommendations from the 2001 evaluation, RCN decided in 2003 to use foresight as an integrated element in planning a new type of strategic instrument: the so-called large-scale programmes. The rationale was that “by utilizing a scenario- based foresight methodology and emphasizing broad participation, the quality of strategic plans and program development processes would be enhanced.”¹⁴ RCN launched a set of interconnected foresight activities, spanning roughly the years 2003- 2007. Now that several of the large programmes are coming to an end, we see similar initiatives being launched to prepare for the design of the follow-up programmes.

RCN adopted foresight based on systematic study. A background discussion on foresight studies was provided through the 2003 CREATE report that RCN commissioned in 2003. This included an inventory of foresight studies. Consultancies and public bodies dominated the list, but it also included a few industry players. The most common type of foresight study in Norway was the use of scenarios, and the CREATE project suggested these were commissioned ‘out of interest’ or as a basis for decision-making. Largely, foresight studies tended to focus on national challenges – oil and energy, marine industries and environment inter alia – with a handful of the projects focusing on European or international projects. Studies were identified as:

societal or culturally themed; industry-oriented; technology-oriented; or theory- oriented foresights.

In parallel with this initiative, the Research Council funded five pilot foresight studies – Biotek Norge 2020, Avanserte materialer Norge 2020, Havbruk 2020, Energi 2020+ and Utsikt. These were produced with the aim of gaining experiences of foresight and dialogue-based methods as a tool in RCN programme planning and strategy processes.

A research policy commentary on the five thematic foresights¹⁵ concluded that the foresight exercises had been successful in achieving their original goal of competence development, and were generally seen as positive experiences which brought more, and a broader set of, participants to the table and was said to have received an increased amount of public attention. They generated new ideas and created new networks and arenas. They also raised internal competence within RCN. The foresights projects exposed some shortcomings regarding the process of utilization of the results in the programme development and strategy processes that they were intended to support.

The report recommended that RCN (i) develop a strategy for future collaborative work involving ‘horizontal partnerships’ (ii) develop its capacity for development work, (iii) build on the competence developed and create partnerships for future collaborative work, and avoid building up an internal foresight expertise.

In October 2007, the Research Council and the Norwegian Institute for Urban and Regional Research (NIBR) organised a regional foresight conference. This had as a goal to articulate input to current challenges and experiments in foresight studies. The conference also looked in details at the conditions for regionalisation and cluster development, as well as ways in which to develop the knowledge base through programmes like VRI and Arena¹⁶.

12 Årsrapport 2007 – Del III, p.156 13 Årrsrapport 2008 – Del III KD, p.183

14 Egenvurdering av satsingen på foresight og dialogbaserte arbeidsformer i Norges forskningsråd 2003 til 2005, Norges Forskningsråd 2006

15Trenger vi nye former for tverrfaglighet og samspill? En forskningspolitisk kommentar til fem foresightprosjekter, Norges Forskningsråd 2006

16 Nye framtider i regionene, Norsk institutt for by- og regionforskning and Norges Forskningsråd, 2007

(18)

Since 2007, RCN has made little use of formal foresight. However, in preparation for the second generation of the Large-scale programmes, RCN has conducted various large-scale stakeholder consultations, receiving input from multiple research communities and providing the opportunity for vast stakeholder input through web applications. An illustration is the process adopted in the Biotek 21 initiative.

BIOTEK 2012

Following up on the bioscience evaluation in 2000 and an initiative of the research community, in 2002 RCN launched the national research programme for functional genomics (FUGE). In 2004/2005 RCN conducted a consultation process where different actors developed scenarios for the Norwegian biotechnology ("Biotech Norway 2020"). The results of this exercise fed into the work plans for the FUGE large-scale programme.

The FUGE programme ended in 2011 and for the design of the follow-up programme, an open and inclusive consultation process was launched, "BioTek 2012". The objective was to identify areas where Norwegian biotechnology has the opportunity to contribute to solving social challenges and to strengthen the national added value. The ambition was to lay the groundwork for a broad community dialogue about the challenges and opportunities related to research, innovation and economic development based in biotechnological methods and knowledge. The initiative also included analyses of a number of national and international policy documents and evaluations. This exercise was carried out in close collaboration with universities, colleges and research institutes and the private sector and other relevant actors. There was a strong predominance of feedback from academia rather than the business community. Two major interest groups represented the private sector, i.e. the Pharmaceutical industry association and the Norwegian Biotech Forum; their comments were primarily focused on human biomedicine.¹⁷

The consultation is to be set within a broader initiative, launched by a set of Ministries for the development of the National Strategy for Biotechnology 2011. That multi- ministry initiative was prepared by the Ministry of Health and Care Services, Agriculture and Food, Fisheries and Coastal Affairs, Trade and Industry, and the Ministry of Environment, under leadership of the Ministry of Education and in collaboration with RCN and Innovation Norway.

2.2.3 The Role and Value of the RCN Meeting Places

The term ‘Meeting Places’ refers to all opportunities created or exploited by RCN, for knowledge sharing with key stakeholder groups, government and research institutions. These include formal meetings with the Ministries (which we also consider in our report on governance), meetings with representatives or groups of research institutes, universities or industry and/or their associations, meetings of RCN’s steering committees and other committees such as the proposal appraisal ones, as well as conferences and events, such as the conference of the International Polar Year.

In the sections below we further describe these meeting places, including the representation of the different stakeholders, and report on their value from the stakeholders’ point of view.

2.2.3.1 Meetings with the research and industry stakeholders

17Rapport fra ”BIOTEK 2012”: Kunnskapsgrunnlag for fremtidig satsing på bioteknologi, RCN, November 2010

(19)

In its 2012 report on the RCN meeting places¹⁸, RCN reports on an internal survey trying to establish the intensity of RCN’s activities from this perspective. In 2011, RCN organised or participated in approximately 350 ‘meeting places’ involving meetings with stakeholder groups or events, excluding the management committees and appraisal/competition panels. This implies that there is a fraction less than one meeting for every single day of the year. The organisational responsibilities for these

‘meeting places’ were widely spread across the hierarchical system and only 14% were co-managed by different ‘units’ (program / activity, department, division management, communication, team). Most active were the Innovation and Health and Society Divisions, least active the Science and Energy, Resources and Environment ones.

Our analysis of the RCN committees showed that in the period 2003-2011, RCN set up a total of 238 committees, involving 1,541 individuals. We categorised these committees in terms of their function and type of input/support provided. As shown in Figure 4, the majority of committees (70%) were programme/instrument boards or scientific discipline committees, supporting RCN in the overall programme management processes; 14% were strategy/programme development committees (e.g.

in charge of the development of research roadmaps).

Especially the Innovation division involved stakeholders to provide input for strategy and programme planning activities, while the Division for Strategic Priorities managing the Large-scale programmes set up 4 committees to foster internationalisation.

Figure 4 Type of committees or boards (2003-2010) (Percentages)

Source: RCN committees database, Technopolis analysis

In terms of stakeholder involvement, in its 2012 report¹⁹ RCN indicated that for the meetings with stakeholder groups, "Researchers" is the dominant target, followed by

"public administration" and "funding ministry." "Business / industry" was the least targeted stakeholder group.

Except in the Innovation Division, university researchers were the category of people most frequently involved in the RCN steering committees (Table 1). As one would expect, the industry sector constituted the most represented stakeholder group in that Division. Industry is little represented in the Strategic Priorities Division’s committees. The Institutes sector is also poorly represented. According to interviewees, this is the case in order to avoid conflicts of interest, especially in the

18 Forskningsrådets møteplasser 2011 og 2012 – Rapport fra kartlegging, RCN, 2012 19 Forskningsrådets møteplasser 2011 og 2012 – Rapport fra kartlegging, RCN, 2012

Steering Board (HS & DS)

2%

Programme management committee

70%

Strategy/

Programme Development Comm.

14%

Int'l cooperation committee

5%

Dissemination committee

3%

Fund management

committee 3%

Evaluation Committee

3%

Steering committees in the RCN, 2003 - 2010 base: 172 committees

(20)

programme boards. It should be noted that the current Executive Board (HS) now has a member from the institute sector – for the first time ever.

There was a considerable decrease over the years in the involvement of public administration (ministries, regions or town councils). In 2009, an agreement was reached that limited the potential membership of Ministries in programme boards to the policy-oriented programmes in order to avoid either the appearance or the reality of ministries steering programmes at the project level.

Table 1 Stakeholder involvement in RCN management committees, per division

Stakeholder group Division for

Strategic Priorities Innovation

Division Science

Division Grand Total

University 24% 15% 54% 33%

Industry Sector 7% 38% 4% 16%

Foreign experts 17% 5% 24% 16%

Institute Sector 15% 10% 7% 10%

National PA 16% 9% 3% 8%

Public Agency 11% 6% 2% 6%

Univ. Colleges 4% 4% 3% 4%

Other 4% 3% 1% 3%

Regional/Local PA 2% 4% 1% 2%

N.A. 0% 5% 1% 2%

Total 100% 100% 100% 100%

Total number of participations 474 573 693 1,740

Women are quite well represented in the committee system. The committees contained on average 44% of women (2003-2010). Female experts constitute about 54% of the members of international cooperation committees and are fairly well represented also in dissemination committees and steering boards (about 49% and 46% respectively). However, they are underrepresented in ‘strategic’ committees (programme design and road mapping committees).

Over the years, regional representation in RCN committees has improved. Over the whole period (2003-2011), the majority of participants were from individuals active in institutions or firms based in the capital region (Hovedstaden) but their relative importance has decreased and experts based in other regions are now more frequently involved, in particular those based in West Norway (Vestlandet).

Figure 5 Regional participation in RCN committees, 2003-

Source: RCN Committees database. Technopolis analysis, 2012

Hovedstaden;

43%

Vestlandet; 13%

Midt-Norge; 12%

Nord Norge; 9%

Oslofjordfondet;

3%

Innlandet; 2%

Agder; 1%

Foreign experts;

15%

n.a.; 2%

Participations at the regional level 2003-2010, base: 1,817 participations

(21)

2.2.3.2 The stakeholders’ perspective

Stakeholders consulted in the survey conduced in the context of this study were overall rather sceptical about the effect of their participation in the committees on RCN’s funding schemes or policy/processes (Table 2). Curiously, the responses of members and past members of RCN Boards varied little from the average figures shown in the Figure. We would have expected them to feel they had more influence than others over RCN policies, schemes, procedures and processes.

Table 2 Meeting place function: views on results from participation in RCN ‘strategy meetings’

To what extent did your participation result in:

Very large

Large Moderate Limited Not at all

Cannot say

N=

Your improved understanding of the rationale for RCN policies and strategies

9.4% 36.4% 34.1% 11.8% 2.0% 6.3% 651

Your improved insight into a wider set of research areas

6.4% 35.3% 33.9% 14.6% 3.6% 6.3% 638

Input to RCN for changes in

policies/strategies 3.6% 22.5% 36.3% 18.8% 8.1% 10.8% 640

Input to RCN for changes in funding schemes

2.2% 10.3% 30.6% 23.7% 17.1% 16.1 633

Input to RCN for changes in

management/procedures 0.9% 8.5% 23.6% 25.6% 22.9% 18.5% 542

Changes in RCN policy or processes 0.6% 7.1% 22.% 26.5% 23.7% 19.2% 634 Source: NIFU survey of leaders of Norwegian research institutions, researchers and participants in RCN meeting places (WP5)

Interviewees that had been involved in programme boards expressed their frustration at how little account RCN staff can take into their suggestions for changes in programmes because of lack of flexibility in the of system. They nevertheless did consider that programme boards and their inputs are of considerable value for RCN staff and for themselves in terms of knowledge building. Especially Division Board members emphasised this aspect, though they tended to mention it as a sort of compensation for their DS’ lack of power.

Similarly, a reasonably high proportion of stakeholders consulted in the survey said they gained personally from their participation in the committees in terms of an improved understanding of the rationale for RCN policies and strategies and an improved insight into a wider set of research areas.

According to a large majority of survey respondents, RCN has an important role in the Norwegian system as ‘meeting place’ for policy discussions; for respondents from the research sector in particular in relation to research policies, for those from trade and industry and the government/public sector in relation to innovation policies.

As part of the analysis of its activities related to strategic intelligence, reported in the 2011 document “Knowledge-based advice, tools and meeting places - Policy for RCN’s work in the knowledge base”²⁰, RCN also asked for feedback from the Ministries on the quality and relevance of its advisory activities. In general, the feedback was positive. Several ministries considered that RCN has been good at organising and developing venues and actively uses conferences and workshops to develop a broad knowledge base. The Ministry of Health, however, pointed out that they would welcome wider involvement in dialogue arenas, for example from the voluntary sector.

20 Kunnskapsbaserte råd, virkemidler og møteplasser - Policy for Forskningsrådets arbeid med kunnskapsgrunnlaget – med fokus på det tverrgående kunnskapsgrunnlaget, RCN, 2011

(22)

2.3 Evaluation as strategic intelligence

2.3.1 Evaluation Strategy and Implementation

RCN is the main body responsible for research evaluations in Norway. The statutes say that RCN should “ensure the evaluation of Norwegian research activities”²¹, which is a slight change of wording from the earlier stattes, which require the Council to “initiate and follow-up evaluations of research and research-performing institutions”²². RCN describes its evaluation activities using five categories.

• Scientific discipline evaluations, which are deemed important channels of communication with the Norwegian HEI sector. The RCN science division has a five-year plan for these types of study, which also encompass research conducted at research institutes and hospitals

• Evaluations of instruments. These are predominantly programme evaluations, looking at outcomes and impacts of R&D programmes or other RCN activities, programme design and goal achievements

• Evaluations of institutes. RCN has stopped doing these on the understanding that their direction is replaced by the new core funding system

• Other types of evaluation. These are generally contracts between RCN and particular ministries and tend to focus on policy reforms relevant to social policy and higher education, where RCN’s expertise in procurement and research administration is exploited

• Self-evaluations, predominantly undertaken by the Innovation Division in RCN.

This type of evaluation is implemented using a model developed by TAFTIE (The European Network of Innovation Agencies), and is focused around the improvement of additionality in innovation programmes.

This section focuses on RCN’s commissioning and use of evaluations in the last decade. It looks in-depth into a number of RCN programme and instrument evaluations, discipline-specific evaluations and evaluations of research institutes.

2.3.2 Background

In 2001, we criticised the decentralisation of the evaluation budget, which lessened the influence and effects of evaluations studies and suggested that centralisation of the evaluation function would strengthen the strategic use of evaluations. At that point, RCN had two evaluation cycles in place.

• A six-year cycle of evaluation for the institutes under its auspices, which were evaluated by peer review, accompanied by user surveys

• A discipline evaluation cycle, managed by the Research Council’s NT division.

We found that, with the exception of the discipline evaluations, there was an overall lack of organised and systemic use of evaluation studies within the Research Council.

Similarly, there was no systematic evaluation training for RCN staff, beyond ad-hoc activities. Evaluation activities lacked sufficient consequence with regards to policy or learning. This was in particular true for institute evaluations, as RCN had little influence on funding or core funding levels. RCN was making extensive use of peer reviews. We indicated that in particular when evaluating larger entities such as disciplines, groups of institutes and policies, the use of other techniques

21 Statutes of the Research Council of Norway, New version – 1 January 2011

22 Erik Arnold, Stefan Kuhlman and Barend van der Meulen, A Singular Council Evaluation of the Research Council of Norway, Oslo: Royal Norwegian Ministry of Education, Research and Church Affairs, 2001

Evaluation of the Research Council of Norway

31 August 2012

Evaluation of the Research Council of Norway

Background Report No 1. - Production of Strategic Intelligence and Advice

Evaluation of the Research Council of Norway

Background Report No 1. Production of Strategic Intelligence and Advice

Table of Contents

Table of Figures

List of tables

Summary

1. Introduction

2. Strategic Intelligence