Test solve exams
with LLMs
Saverio Bolognani
LLMs can easily solve our exams.
Let's not talk about that.
| Otherwise... | |
|---|---|
| Clarity | Ambiguous interpretation, answers are not informative |
| Time | Unanswered questions (also: cultural expectation) |
| Alignment | Out-of-scope competences determine the grade |
| Coverage | Correlated questions add little information on the student's level |
what makes a good written exam
Standard procedure: test-solve the exam
LLM-based test solving - what do you need?
students
Who are the students, what do they know already, what is their level
Data: course catalog, your input, past questionnaires
Competences and course content
Data: teaching material, syllabus, your input
course
Your exam draft
Data: LaTeX draft
exam
Prepare a STUDENTS.md file that describes the level, compentences, pre-existing knowledge, and skills of a student that enrolls in the COCO course. Do not include competences and knowledge that they acquire during the COCO course. I will provide you multiple sources of information for this task.
+ + +
Ask me questions to refine the description, if some uncertainty is left
Students of the course are first or second year master student at ETH Zurich, the all have a bachelor level knowledge of linear algebra and calculus.
They have all taken a Control System 1 course that covers....
Prepare a
COURSE-CONTENT.md
file that describes the content covered in the course and the competences that the students need to learn.
I will provide you multiple source of information for this task.
+ + +
Ask me questions to refine the description, is some uncertainty is left
Even better: split into
- COURSE-COMPETENCES.md
- COURSE-CONTENT.md
- EXAM-MODALITY.md
- ...
A good idea in general
Helps you prepare better exams and tutorials, focus on what matters.
ETH Learning to teach course or equivalent
Consider students described in STUDENTS.md that just took the course COURSE-CONTENT.md. Solve the exam, write the solutions in a separate LaTeX file, and perform this analysis.
- Estimate the time needed to complete the exam for an average student and for an excellent student.
- Are there questions that could be interpreted ambiguously?
- Are there questions that assess the same competence and knowledge or have significant overlap?
- Are there questions whose correct answer is heavily determined by competences and skills that are not part of the course?
- Find bottlenecks: mistakes that would affect the score of multiple questions.
TEST SOLVE!
- What is the question that is most likely to determine the final score of the exam?
- Are there questions that most students will answer and that will not provide useful information on the student's level?
- Is there significant risk that a student will have to attempt multiple strategies to solve some of these questions? How much time would be wasted?
- Is there any specific knowledge whose memorization is not part of the competence goals and that should be included in the exam?
- Estimate the points that each exercise should be worth, proportional to the effort required, sum to 100.
- How many blank pages will students need for this exam?
Write the findings in FINDINGS.md / chat about this / comment the LaTeX file / propose fixes / ...
Time available for the exam
Tasks were clear and understandable
Alignment of content and difficulty