Today is an experiment

https://2029.ref.ac.uk/about/timetable/

https://2029.ref.ac.uk/guidance/

https://2029.ref.ac.uk/guidance/section-4-contributions-to-knowledge-and-understanding-cku-guidance/

Overview

• 4 main panels
• • A: Medicine, Health and Life Sciences (UoAs 1 to 6)
  • B: Physical Sciences, Engineering and Mathematics (UoAs 7 to 12)
  • C: Social Sciences (UoAs 13 to 24)
  • D: Arts and Humanities (UoAs 25 to 34)
• 34 Units of Assessment (UoAs) and 34 sub-panels
• 2 advisory panels 
  • People and Diversity Advisory Panel: supporting recognition of a diversity of roles and careers
  • Research Diversity Advisory Panel: supporting equitable recognition of diverse forms of research

What is assessed?

• People, culture and environment (PCE)

  • 25% weighting

  • replaces the environment element and will include an increased emphasis on research culture

• Contribution to knowledge and understanding (CKU)

  • 50% weighting

  • replaces the outputs element. It will largely be based on assessment of research outputs and will also include evidence of broader contributions to the advancement of the discipline

• Engagement and impact

  • 25% weighting

  • replaces the impact element. It will consist of impact case studies and an accompanying statement

How are outputs assessed?

• Originality
  • the extent to which the output introduces a new way of thinking about a subject, or is distinctive or transformative compared with previous work in an academic field
• Significance
  • the extent to which the work has exerted, or is likely to exert, an influence on an academic field or practical applications
• Rigour
  • the extent to which the purpose of the work is clearly articulated, an appropriate methodology for the research area has been adopted, and compelling evidence presented to show that the purpose has been achieved

What outputs are eligible?

5.6.5. A glossary of output categories and collection formats will be published.  Further guidance on types of outputs will be published later in the Autumn in an update to REF 2021 Guidance on Submissions Annex K, which offers a good indication of the potential range of outputs.

Potential new outputs

5.6.4. Examples of diverse output types include, but go well beyond, journal articles, conference papers, monographs and book chapters, to include, as appropriate to each UoA:

• non-textual (artefacts, audio, visual, multi-modal)
• creative, performance and practice-led outputs
• reviews and research synthesis
• software and code
• datasets
• translations and critical editions
• reagents
• multi-component outputs
• policy summaries and analyses
• different types of reports

What's the difference between assessing NTOs and TOs?

5.6.7. An underpinning principle of the REF is that all forms of research output and research practice will be assessed on a fair and equal basis. Sub-panels will not regard any particular form of output as of greater or lesser quality than another per se, and all subject matter will be assessed equally, without bias.  

Which outputs will we use?

ESS Round 8 (34,033 user downloads) was developed and produced over three years under the coordination of ESS HQ (Harrison, co-director). It contains approximately 40,000 cases from 23 countries and 534 variables. Documented in a 190-page report, Round 8 covered 11 topics. Using strict random probability sampling and hour-long face-to-face interviews, it included core and rotating modules on public attitudes to climate change, energy preferences and welfare attitudes. ESS HQ organised the call for and selection of rotating modules, a lengthy period of question design, development, pre-testing and administration in the field, data cleaning, deposit and release.

NTOs for assessment

1. Discuss the NTOs in your group

2. Decide how to rank them on the basis of Significance

3. Make notes about Originality and Rigour

4. Make notes about which decisions were straightforward and which were not

5. Make notes about what information is missing and report back

​Notes at: https://tinyurl.com/NTOLiverpoolEx2

Exercise 2:

Housekeeping

1. Before starting the conversation...
2. Choose a chair who will ensure that everyone’s voice is heard and will report back at the end of the session
3. Choose a note taker who will be responsible for taking notes about any significant thoughts or decisions.

Thank you!

@sjh5000

ORCID: 0000-0002-6809-5195, EPSRC Grant no: EP/S021779/1

Licence

 © Simon Hettrick. These slides are licensed under a Creative Commons Attribution 4.0 International : https://creativecommons.org/licenses/by/4.0/.

Link to slides

https://slides.com/simonhettrick/talk-2-nto-workshops-liv-june-2025

Image credits

• Guinea pigs by Karlijn Prot on Unsplash