ERIC Forum Position Paper on the Development of KPIs for Research Infrastructures 

Research infrastructures (RI) represent public investments. As such, they need to be accountable to their funders by demonstrating expected performance and impact. In its conclusions on “Accelerating Knowledge Circulation in the EU”, the Council of the European Union:

“INVITES Member States and the Commission within the framework of ESFRI to develop a common approach for monitoring of their performance and INVITES the Pan-European Research Infrastructures, on a voluntary basis, to include it in their governance and explore options to support this through the use of Key Performance Indicators.”[1]

The ERIC Forum gladly takes up the Council’s invitation in that it

• Seeks to actively collaborate with ESFRI by connecting with the ESFRI Working Group (WG) on Monitoring, by responding to their questionnaire and proactively developing a common approach in discussing the monitoring system and KPIs,
• Contributes to the development of a common approach and measures for monitoring of their performance via the ERIC Forum Implementation Project [2],
• Includes and further develops Key Performance Indicators (KPIs) in the governance of its members,

While the need for evaluation is widely recognised, there is no firm agreement on the measures that might be used.The term KPI is often used to mean a management tool primarily aimed at internal monitoring of performance on a relatively short timescale. In the case of RIs it is more useful to think of performance metrics and qualitative case studies aimed more at external reporting to key stakeholders, and monitored over longer timescales, particularly where factors such as socio-economic impact are concerned. Success does require an understanding of both a baseline and a trend, but growth is not necessarily always the ideal scenario as it differs from progress.

With this position paper, the ERIC Forum [3] aims to contribute to the development of a framework of performance measures and help to align parallel initiatives. The ERIC Forum appreciates the openness of the ESFRI WG to collaborate on this matter and would like to thank the WG for the opportunity to get involved and discuss details of their work in meetings organized in Brussels (April 2019) and Oslo (May 2019) and for sharing preliminary results. The ESFRI WG has indicated that it would like to invite the ERIC Forum to contribute to the prioritisation of KPIs which will be the outcome of their work. The members of the ERIC Forum would be pleased to participate in such process. The ERIC Forum looks forward to continued collaboration with the ESFRI WG and all involved actors in a constructive way.

ERIC Forum Positions

The ERIC Forum supports the development of a common understanding of KPIs and wants to actively contribute to the ongoing discussions

ERICs understand the need of their funders, i.e. EU member states (in some cases also countries outside the EU) and other stakeholders, to create a framework of performance measures to monitor both the performance and impact of RIs and support an informed decision-making process. The ERICs demonstrated their interest in the development of KPIs by answering the ESFRI questionnaire. The ESFRI Forum Implementation Project dedicates a Work Package to the monitoring, scientific evaluation, and socio-economic impact assessment of ERICs, which can feed into the work of ESFRI WG. The ERICs would like to be involved in the ESFRI process of developing KPIs for RIs, in offering suggestions for consideration. The ERIC Forum appreciates the openness of ESFRI to collaborate on this important matter.

 

KPIs should be tailored to the specific objectives and mission of each ERIC

Every ERIC and RI is unique in terms of its raison d’être, objectives, mission and vision. Even ERICs active in the same scientific domain differ. To be properly meaningful, any metric needs to be customised to the unique character of each RI. ERICs acknowledge that ESFRI is aware of the unique nature of each ERIC and RI. Roadmap 2018 mentions the following: “To help address the unique aspects of each RI, Landmarks will develop specific KPIs and report on the few generic ones chosen by ESFRI.”[4] As ERICs have a good understanding of their own mission and objectives, the ERICS ought to be involved in the selection of generic KPIs.

 

KPIs should only be used to benchmark an RI against its own performance and not to compare RIs

KPIs recommended by ESFRI should provide a common ground for a monitoring system that should later be tailored to the specific needs of each ERIC. KPIs cannot and should not be used to compare one RI against another because their domains, objectives, activities etc. are heterogenous. Even KPIs with the same names often have fundamentally different meaning for different RIs. Many metrics only make sense when tracked over a several year period. This should be taken into consideration when evaluating data. Success does require an understanding of both a baseline and a trend, but growth is not necessarily always the ideal scenario as it differs from progress.

 

KPIs should comply with well-proven criteria for setting up indicators and measures

KPIs should be defined according to one of internationally recognised standards for developing indicators and measures, such as e.g. RACER (Relevant, Acceptable, Credible, Easy and Robust) criteria developed as part of the European Commission’s Impact Assessment Guidelines, SMART (Specific, Measurable, Achievable, Relevant, Time-bound) criteria, CREAM (Clear, Relevant, Economic, Adequate, and Monitorable) etc. Because of their strong connection to the European Commission and their relevance for the European Research Area, ERICs are in favour of RACER criteria. Each ERIC should be allowed to set RACER targets for the recommended KPIs. In most cases, this process will involve their governing bodies, thus implying the need for a flexible timeframe.

 

KPIs and indicators to measure socio-economic impact are not the same, even though a limited number of selected KPIs could be used to measure impact

The purpose of KPIs is to measure performance, evaluate success in delivering results, and monitor progress towards set goals.[5] The purpose of indicators assessing socio-economic impact is to evaluate how RIs transform their environment and what influence they have beyond scientific results. Some RIs enable science by producing data which are used for scientific research or by providing access to state-of-the-art instrumentation, so definition of impact can vary across RIs. Recognising the difference between KPIs and impact indicators, the ERIC Forum Implementation Project has separate tasks dedicated to each of the tools. A KPI framework should exclude indicators to assess socio-economic impact of RIs. KPIs are considered an internal management tool and thus it is more useful to think of performance metrics and qualitative case studies aimed more at external reporting to key stakeholders and monitored over longer timescales.

Conclusions

The performance of ERICs can be demonstrated through qualitative as well as quantitative results. This should be considered in the selection of common KPIs. In the case of RIs it is more useful to think of performance metrics aimed more at external reporting to key stakeholders and monitored over longer timescales.

The activities and objectives of ERICs differ even if they operate in the same scientific area. As a result, KPIs need to be customised to the unique character of each RI. As an internal management tool, KPIs are used to evaluate the performance of RIs on individual basis and should not be used to compare ERICs. The objectives of each ERIC have to be central to the evaluation process, utilising a mixture of methods, measures and indicators that closely relate to the mission of the specific ERIC and its activities.

A timeline for the adoption of common KPIs is as important as their selection. The timeline should be decided with care in order to allow for proper and systematic collection of data.

The ERIC community wishes to be actively involved in the ESFRI process and to engage in the discussion. The ERIC Forum appreciates the openness of the ESFRI WG to collaborate and seek feedback from the ERIC Forum. To contribute to further discussions on this matter, the ERIC Forum will, among other things, disseminate results of activities foreseen in the framework of the EU-funded ERIC Forum Implementation Project. Should a common set of KPIs be defined as an outcome of the ESFRI questionnaire, ERICs and RIs who will be invited to adopt them on a voluntary basis ought to be informed about the purpose of these KPIs and how and by whom they will be used. Members of the ERIC Forum would be happy to review the list of KPIs suggested by ESFRI WG and contribute to their prioritisation.

An individual baseline needs to be established for each RI. Many metrics only make sense when measured over a certain period of time. As a result, an assessment should only be made after several years (2-5) when progress relative to the set baseline can be assessed. After that, individual recommendations for each RI should be made to allow for improvement of their performance, should there be a need.

The ERIC Forum

The ERIC Forum is a collective initiative of currently 23 European Research Infrastructure Consortia (ERICs) and aspiring ERICs. It enhances collaboration and knowledge transfer between ERICs and provides them with a common voice for issues concerning ERICs as part of the European Research Area and the further development of the ERIC regulation. The implementation project of the ERIC Forum is financed by the European Union Framework Programme for Research and Innovation, Horizon 2020.

 

[1] Conclusions of the Competitiveness Council, 29 May 2018, n19.

[2] Horizon 2020 ERIC Forum Implementation Project, grant agreement 823798.

[3] Including aspiring ERICs.

[4] Roadmap 2018: Strategy Report on Research Infrastructures. (ESFRI, 2018). Available at: http://roadmap2018.esfri.eu/strategy-report/the-esfri-methodology/.

[5] For a distinction between KPIs and impact indicators see OECD Science, Technology and Industry Policy Paper no. 65 from March 2019: Reference Framework for Assessing the Scientific and Socio-Economic Impact of Research Infrastructures, pp. 12-13. Available at: https://doi.org/10.1787/3ffee43b-en.