Overview

1. Why we're conducting the evaluation

2. How we're implementing the evaluation

3. Details about the process

4. A wider study across academia

5. Goal for today

The issue

The Institute is a highly distributed organisation:

• Works over five teams

• With staff at four universities

• With communities of thousands

• Across most, if not all, disciplines

• Over a significant number of countries

• Over the last nine years

How do we keep track of our work and maximise the return on our investment?

History

• Measuring the success of the Software Sustainability Institute (2014): http://bit.ly/2lhHM1F


• Reporting back to the Advisory Board (every 6 months)
   
• Metric collection during bidding process (twice since 2010)

Goals

1. Decide on a more reliable method of measuring the progress of the Institute's work
    

2. Report metrics at each Advisory Board meeting
    

3. Have the system in place by the mid-term of the current grant when we are re-assessed
    

4. Achieve the above goals with the minimum investment of effort and resources... more on this later

Building a framework

• Employ an expert to construct to help us evaluate ourselves

1. What particular challenges does SSI address?

2. How will the world be different if SSI is successful?

3. How do you think SSI is thought of now (and by whom?)

4. How would you like SSI to be thought of 5 years from now (and by whom)?

Building a framework

The evaluation framework was built via

• Initial scoping meetings to identify the vision and goals of the Institute

• Meetings with all senior staff

• Individual meetings with senior staff

• Literature review of previous Institute documentation

• Produce and agree objectives

• Development of SSI Impact Evaluation Framework

Definitions

Impact: how the world will be changed by the endeavour (4 of these)

Outcome: A change in understanding or behaviour (2-3 of these per impact)

Outcome indicator: A way of measuring the change in an outcome (1-6 of these per outcome)

Achieving balance

Time spent on metric collection is time not spent on supporting the research community

For each indicator, we must balance:

• How compelling it is
• Ease of collection
• Availability of the data

Outcome

"Researchers engaging with software will have a basic level of awareness of best practice concerning development of software and know where to go to get further support"

Outcome indicator

"Usage of relevant SSI online resources (& help desk) increase following outreach & training events"

Example 1

Example 2

Outcome

"There will be career paths across the full range of seniority for people who contribute to research through the creation of software (e.g. “research software engineers”)."

Outcome indicator

"Number of institutions with job titles for "research software engineer"

Example 3

Outcome

"Funding agencies provide resources for software development, preservation, and archiving"

Outcome indicator

"A diverse set of funding agencies provide resources for software development, preservation, and archiving"