ERIC Forum Position Paper on the Development of KPIs for Research Infrastructures
Research infrastructures (RI) represent public investments. As such, they need to be accountable to their funders by demonstrating expected performance and impact. In its conclusions on “Accelerating Knowledge Circulation in the EU”, the Council of the European Union:
“INVITES Member States and the Commission within the framework of ESFRI to develop a common approach for monitoring of their performance and INVITES the Pan-European Research Infrastructures, on a voluntary basis, to include it in their governance and explore options to support this through the use of Key Performance Indicators.”[1]
The ERIC Forum gladly takes up the Council’s invitation in that it
- Seeks to actively collaborate with ESFRI by connecting with the ESFRI Working Group (WG) on Monitoring, by responding to their questionnaire and proactively developing a common approach in discussing the monitoring system and KPIs,
- Contributes to the development of a common approach and measures for monitoring of their performance via the ERIC Forum Implementation Project [2],
- Includes and further develops Key Performance Indicators (KPIs) in the governance of its members,
While the need for evaluation is widely recognised, there is no firm agreement on the measures that might be used.The term KPI is often used to mean a management tool primarily aimed at internal monitoring of performance on a relatively short timescale. In the case of RIs it is more useful to think of performance metrics and qualitative case studies aimed more at external reporting to key stakeholders, and monitored over longer timescales, particularly where factors such as socio-economic impact are concerned. Success does require an understanding of both a baseline and a trend, but growth is not necessarily always the ideal scenario as it differs from progress.
With this position paper, the ERIC Forum [3] aims to contribute to the development of a framework of performance measures and help to align parallel initiatives. The ERIC Forum appreciates the openness of the ESFRI WG to collaborate on this matter and would like to thank the WG for the opportunity to get involved and discuss details of their work in meetings organized in Brussels (April 2019) and Oslo (May 2019) and for sharing preliminary results. The ESFRI WG has indicated that it would like to invite the ERIC Forum to contribute to the prioritisation of KPIs which will be the outcome of their work. The members of the ERIC Forum would be pleased to participate in such process. The ERIC Forum looks forward to continued collaboration with the ESFRI WG and all involved actors in a constructive way.
ERIC Forum Positions
The ERIC Forum supports the development of a common understanding of KPIs and wants to actively contribute to the ongoing discussions |
|
KPIs should be tailored to the specific objectives and mission of each ERIC |
|
KPIs should only be used to benchmark an RI against its own performance and not to compare RIs |
|
KPIs should comply with well-proven criteria for setting up indicators and measures |
|
KPIs and indicators to measure socio-economic impact are not the same, even though a limited number of selected KPIs could be used to measure impact |
|
Conclusions
The performance of ERICs can be demonstrated through qualitative as well as quantitative results. This should be considered in the selection of common KPIs. In the case of RIs it is more useful to think of performance metrics aimed more at external reporting to key stakeholders and monitored over longer timescales.
The activities and objectives of ERICs differ even if they operate in the same scientific area. As a result, KPIs need to be customised to the unique character of each RI. As an internal management tool, KPIs are used to evaluate the performance of RIs on individual basis and should not be used to compare ERICs. The objectives of each ERIC have to be central to the evaluation process, utilising a mixture of methods, measures and indicators that closely relate to the mission of the specific ERIC and its activities.
A timeline for the adoption of common KPIs is as important as their selection. The timeline should be decided with care in order to allow for proper and systematic collection of data.
The ERIC community wishes to be actively involved in the ESFRI process and to engage in the discussion. The ERIC Forum appreciates the openness of the ESFRI WG to collaborate and seek feedback from the ERIC Forum. To contribute to further discussions on this matter, the ERIC Forum will, among other things, disseminate results of activities foreseen in the framework of the EU-funded ERIC Forum Implementation Project. Should a common set of KPIs be defined as an outcome of the ESFRI questionnaire, ERICs and RIs who will be invited to adopt them on a voluntary basis ought to be informed about the purpose of these KPIs and how and by whom they will be used. Members of the ERIC Forum would be happy to review the list of KPIs suggested by ESFRI WG and contribute to their prioritisation.
An individual baseline needs to be established for each RI. Many metrics only make sense when measured over a certain period of time. As a result, an assessment should only be made after several years (2-5) when progress relative to the set baseline can be assessed. After that, individual recommendations for each RI should be made to allow for improvement of their performance, should there be a need.
The ERIC Forum
The ERIC Forum is a collective initiative of currently 23 European Research Infrastructure Consortia (ERICs) and aspiring ERICs. It enhances collaboration and knowledge transfer between ERICs and provides them with a common voice for issues concerning ERICs as part of the European Research Area and the further development of the ERIC regulation. The implementation project of the ERIC Forum is financed by the European Union Framework Programme for Research and Innovation, Horizon 2020.
[1] Conclusions of the Competitiveness Council, 29 May 2018, n19.
[2] Horizon 2020 ERIC Forum Implementation Project, grant agreement 823798.
[3] Including aspiring ERICs.
[4] Roadmap 2018: Strategy Report on Research Infrastructures. (ESFRI, 2018). Available at: http://roadmap2018.esfri.eu/strategy-report/the-esfri-methodology/.
[5] For a distinction between KPIs and impact indicators see OECD Science, Technology and Industry Policy Paper no. 65 from March 2019: Reference Framework for Assessing the Scientific and Socio-Economic Impact of Research Infrastructures, pp. 12-13. Available at: https://doi.org/10.1787/3ffee43b-en.