Measuring socio-economic impact

The socio-economic impact (SEI) of Research Infrastructures remains a challenging topic. The approaches to assessing SEI have been varied, and a number of methodologies have been developed in the last few years, for example, OECD and a number of HORIZON2020 projects (see below for more info.) Many ERICs have utilised the approaches, and have also used external companies to conduct the assessment.

ERIC’s SEI differs between the different scientific domains, and it can also differ from the preparatory phase to the operational phase, depending on the purpose of the infrastructure. The idea of assessing the SEI of an ERIC needs to be reflected against the life cycle of the ERIC.

ERIC’s performance can be defined as the result of its activities (such as collecting data) and the subsequent output (such as scientific publications). The SEI, then, would be the direct outcome and the longer-term effects of these – for example when policy-makers use published results that are based on the RI-generated data, and this in turn supports actions towards changes in policies or practices.

Finding the links between the ERIC’s activities and the actual SEI that results from them can be challenging. While some activities can produce SEI that is easier to track short-term, others’ SEI may take a long time to materialise, making the trackability and creditability difficult. The impact is often a part of a holistic observation and chain of events.

The starting point in attempting to define and assess an ERIC’s SEI is the ERIC’s strategy. It is useful to ask the following questions:

1. What is the ERIC’s overall mission, and what are its overall objectives?
2. What is the ERIC expected to produce, perform or enable, and how can this performance be measured?
3. What are the short- and longer-term effects and impacts that are expected to materialise from the ERIC’s performance?

The answers to these questions contribute to the understanding of an individual ERIC’s socio-economic impact. The level of the impact can vary, for example, from an instant solution to a phenomenon to a slowly-evolving change in policy-making practices.

Figure 1. An approach to defining SEI

Because performance and impact are often linked, the same or similar indicators – namely KPIs (Key Performance Indicators) and SEII (Socio-Economic Impact Indicators, also sometimes referred to as Key Impact Indicators) – are often used to monitor both. Using these two indicators together might actually be helpful in the attempts to track the SEI from performance to impact. Linking carefully selected performance indicators (KPIs) to expected impacts makes it more straightforward to define suitable impact indicators. Reflecting on questions like ‘what can we measure with performance indicators that would feed into the aimed-at impact?‘ might be useful. See more on KPIs here. 

It’s also important to remember the expectations of your audience. Stakeholders have specific expectations that vary across scientific, geographical, cultural, and political fields. It is, hence,  useful to consider looking at impact indicators and the ERIC’s different stakeholder- and user groups together to develop indicators that produce purposeful data.

Indicators are often only providing numerical data, while validating both performance and impact  often also requires narrative (qualitative) indicators, such as interview data, success stories, experience narratives, and so on. When combining both qualitative and quantitative indicatorsit becomes easier to build up the ‘impact narrative’ or ‘impact landscape’ that is developing throughout the RI’s lifecycle.

ERIC Forum has conducted a survey on SEI assessment practices among 21 ERICs and 11 prep-ERICs. Respondents conveyed several points of view on the importance of assessing the SEI of their ERICs and used several different strategies in carrying out their own assessments.

Four main approaches to impact assessment were discovered:

1. SEI assessment is either connected to Management Plan and/or evaluation process or done through KPIs
2. SEI assessment is done using available tools (Accelerate, RI-Paths etc.) with internal resources
3. SEI assessment is done using a mix of internal and external resources (available tools used as guidelines internally)
4. SEI assessment is carried out in connection with an EU-funded project

While some challenges were the same for all ERICs, respondents were also able to define domain-specific challenges in assessing SEI.

The full ERIC Forum survey on SEI assessment practices can be accessed in the D4.3 Report on SEI ERIC Framework. 

Watch an ERIC Forum webinar “Socio-economic impact of ERICs” below.