LLM + EXAMS

NCCR Automation Retreat 2026

1. Exam test-solving and constructive alignment
   (interactive)
    
2. Personalized exam questions
   (demo only)

what makes a good written exam

Standard procedure: test-solve the exam

LLM-based test solving

students

Who are the students, what do they know already, what is their level

Data: course catalog, your input, past questionnaires

Competences and course content

 

Data: teaching material, syllabus, your input

course

Your exam draft


 

Data: LaTeX draft

exam

Pick a course you have been teaching / TAing + Claude code / Codex (or web chat)

Prepare a STUDENTS.md file that describes the level, compentences, pre-existing knowledge, and skills of a student that enrolls in the COCO course. Do not include competences and knowledge that they acquire during the COCO course. I will provide you multiple sources of information for this task.

+ + +

Ask me questions to refine the description, if some uncertainty is left

Prepare a
COURSE-CONTENT.md
file that describes the content covered in the course and the competences that the students need to learn.

I will provide you multiple source of information for this task.

+ + +

Ask me questions to refine the description, is some uncertainty is left

Even better: split into 

• COURSE-COMPETENCES.md
• COURSE-CONTENT.md
• EXAM-MODALITY.md
• ...

A good idea in general

Helps you prepare better exams and tutorials, focus on what matters.

ETH Learning to teach course or equivalent

Consider students described in STUDENTS.md that just took the course COURSE-CONTENT.md. Solve the exam, write the solutions in a separate LaTeX file, and perform this analysis.

• Estimate the time needed to complete the exam for an average student and for an excellent student.
• Are there questions that could be interpreted ambiguously?
• Are there questions that assess the same competence and knowledge or have significant overlap?
• Are there questions whose correct answer is heavily determined by competences and skills that are not part of the course?
• Find bottlenecks: mistakes that would affect the score of multiple questions.

TEST SOLVE!

• What is the question that is most likely to determine the final score of the exam?
• Are there questions that most students will answer and that will not provide useful information on the student's level?
• Is there significant risk that a student will have to attempt multiple strategies to solve some of these questions? How much time would be wasted?
• Is there any specific knowledge whose memorization is not part of the competence goals and that should be included in the exam?
• Estimate the points that each exercise should be worth, proportional to the effort required, sum to 100.
• How many blank pages will students need for this exam?

Write the findings in FINDINGS.md  /  chat about this  / comment the LaTeX file / propose fixes  /  ...

Time available for the exam

Tasks were clear and understandable

Alignment of content and difficulty

grading project work

COCO project

Vertical Rocket Landing

You are a control expert, and you are asked to act as a consultant for an aerospace company.
This company wants to hear your opinion on their rocket landing control scheme.

You need to produce a 5-slide presentation for the Chief Technology Officer of the company.

Each slide serves a specific purpose:
...
3. Your recommendation: Explain what type of controller you would recommend (among those seen in class). Provide the three most important reasons that support your choice
(list three strengths of the controller and explain how they are achieved). [3 points]
...

2 questions on the presentation that students have to answer at the exam.

• It simulates the questions that they would receive in a presentation.
• It challenges their statements and gives the chance to defend them.
• It tests the depth of the students' work
  • authorship
  • understanding of the problem
  • broader and deeper investigation
• It tests their ability to address technical questions.

experiment: personalized questions

PIPELINE

1. Extract slide 3 of the presentation (Recommendation slide)
2. Pick one of the three reasons provided and challenge it.
3. Format the question according to the exam format.
4. Produce a 1-page document with the slide, the question, and blank space to answer it.
5. Produce PDFs to print.

How to build the question
- Quote 3–8 words verbatim (in quotation marks) from the chosen reason, and where possible cite a concrete specific from their deck (number, named component, figure).
- Ask one fair, pointed question about the assumption the reason rests on, a regime where it may break, a gap between claim and mechanism, or a relevant project-specific test.

- Use concrete project language; prefer wording that describes a physical change or an observable behavior over abstract control jargon. Retain a technical term only when it is itself the course concept being tested, and phrase the rest of the question in plain English. Reuse the student's own concrete terminology where possible. The question must be understandable by students for whom
English is a second language.

Purpose: the question needs to verify whether the student has learned the concepts
listed in the COCO syllabus.

And more!
Test on a few, fine tune, repeat.

The question must sound like a genuine technical question about the
recommendation, not an accusation that the student's work is incomplete. Do
not assume the student performed an additional test that is not explicitly
shown in the deck. When the claimed strength depends on a design choice or
operating range, first ask whether they tried the relevant variation and then
ask what they observed.

Which of the three reasons to pick
- Prefer a reason the student presents as an unqualified strength but which rests on a real, well-known caveat they did not mention.
- Spot mistakes: If present, pick a reason that seems to contrast to the concepts listed in the course syllabus for that specific control method.
- Cross-check slide 5 (Weaknesses). Do not let the question merely restate a limitation the student already lists there — that is too easy. If the natural critique is already on their weaknesses slide, go one level deeper than their stated mitigation, or pick a different reason.

The recommendation slide gives three reasons the student's controller is a
good choice. The question targets exactly one of them with one critique:
a fair challenge that requires the student to defend the reason they listed.

Use at most one short setup sentence followed by exactly one interrogative
sentence (1–2 sentences total, ≤ ~60 words), of the kind asked in the live Q&A right after the presentation. It must be answerable in a few lines by a student who understood and tested their own design. It is not a multi-part exam question.

The purpose is to reveal whether the student can defend the claimed strength,
has thought through its assumptions and limitations, and performed sufficient
project-specific tests. A generic reason copied from standard controller descriptions should be exposed by asking for the mechanism, evidence, or test result that supports it in this project.

Pick one of the three reasons provided and challenge it.

technical tips

 Structure your pipeline for reproducibility. 

Make everything auditable.

Produce a pipeline that runs most things locally

• text extraction
• PDF preparation
• students' data (private)

Reuse and maintain

LaTeX template

auditable instructions

database of questions

DEBATE

1. Are we checking that no LLM is used, or that it is used well?
2. Should we backtrack and revisit what competences we are teaching?
3. Is it a futile exercise? Are they all going to use LLM for similar tasks in the future?
4. Why would a company ask an ETH graduate to provide their expert opinion, as the project said?