Today is an experiment

https://2029.ref.ac.uk/about/timetable/

https://2029.ref.ac.uk/guidance/

https://2029.ref.ac.uk/guidance/section-4-contributions-to-knowledge-and-understanding-cku-guidance/

Overview

• 4 main panels
• • A: Medicine, Health and Life Sciences (UoAs 1 to 6)
  • B: Physical Sciences, Engineering and Mathematics (UoAs 7 to 12)
  • C: Social Sciences (UoAs 13 to 24)
  • D: Arts and Humanities (UoAs 25 to 34)
• 34 Units of Assessment (UoAs) and 34 sub-panels
• 2 advisory panels 
  • People and Diversity Advisory Panel: supporting recognition of a diversity of roles and careers
  • Research Diversity Advisory Panel: supporting equitable recognition of diverse forms of research

What is assessed?

• People, culture and environment (PCE)

  • 25% weighting

  • replaces the environment element and will include an increased emphasis on research culture

• Contribution to knowledge and understanding (CKU)

  • 50% weighting

  • replaces the outputs element. It will largely be based on assessment of research outputs and will also include evidence of broader contributions to the advancement of the discipline

• Engagement and impact

  • 25% weighting

  • replaces the impact element. It will consist of impact case studies and an accompanying statement

How are outputs assessed?

• Originality
  • the extent to which the output introduces a new way of thinking about a subject, or is distinctive or transformative compared with previous work in an academic field
• Significance
  • the extent to which the work has exerted, or is likely to exert, an influence on an academic field or practical applications
• Rigour
  • the extent to which the purpose of the work is clearly articulated, an appropriate methodology for the research area has been adopted, and compelling evidence presented to show that the purpose has been achieved

What outputs are eligible?

5.6.5. A glossary of output categories and collection formats will be published.  Further guidance on types of outputs will be published later in the Autumn in an update to REF 2021 Guidance on Submissions Annex K, which offers a good indication of the potential range of outputs.

Potential new outputs

5.6.4. Examples of diverse output types include, but go well beyond, journal articles, conference papers, monographs and book chapters, to include, as appropriate to each UoA:

• non-textual (artefacts, audio, visual, multi-modal)
• creative, performance and practice-led outputs
• reviews and research synthesis
• software and code
• datasets
• translations and critical editions
• reagents
• multi-component outputs
• policy summaries and analyses
• different types of reports

What's the difference between assessing NTOs and TOs?

5.6.7. An underpinning principle of the REF is that all forms of research output and research practice will be assessed on a fair and equal basis. Sub-panels will not regard any particular form of output as of greater or lesser quality than another per se, and all subject matter will be assessed equally, without bias.  

Which outputs will we use?

NTOs for assessment

1. Discuss the 4 sample NTOs in your group

2. Decide how to rank them on the basis of Significance

3. Make notes about Originality and Rigour

4. Make notes about what information is missing

5. Identify any heuristics (rules of thumb) your group used to make scoring decisions.

Exercise 2:

Heuristics are very interesting

i.e. rules of thumb to simplify and speed up decision-making

Anchor heuristic

The tendency to rely on an initial, or strongly held piece of information (anchor) when making judgements.

Housekeeping

1. Before starting the conversation...
2. Choose a chair who will ensure that everyone’s voice is heard and will report back at the end of the session
3. Choose a note taker who will be responsible for taking notes about any significant thoughts or decisions.

1. Discuss the 4 sample NTOs in your group

2. Decide how to rank them on the basis of Significance

3. Make notes about Originality and Rigour

4. Make notes about what information is missing

5. Identify any heuristics (rules of thumb) your group used to make scoring decisions.

​Notes at: https://bit.ly/48aDaPn

NTOs at: https://tinyurl.com/exeterNTOs

Exercise 2:

Thank you!

@sjh5000

ORCID: 0000-0002-6809-5195, EPSRC Grant no: EP/S021779/1

Licence

 © Simon Hettrick. These slides are licensed under a Creative Commons Attribution 4.0 International : https://creativecommons.org/licenses/by/4.0/.

Link to slides

https://slides.com/simonhettrick/talk-2-nto-workshops-exeter-nov-2025

Image credits

• Guinea pigs by Karlijn Prot on Unsplash