CCSC Northwest 2026

Easy but Unengaging

Rethinking Student Assessments of Instruction with LLM-Driven Dynamic Probing

Jerrick Little, Axel Smart, Lucas Cordova, Heather Kitada Smalley

Willamette University · School of Computing & Information Sciences

The problem

High stakes, an unchanged instrument

Student Assessments of Instruction feed tenure and promotion decisions, yet most still pair a fixed set of Likert items with one or two open prompts. (Hornstein, 2017; Lakeman et al., 2023; Spooren et al., 2013)

What students get

A static form delivered during finals week, with little sign that their feedback produces change. (Vargas-Madriz & Nocente, 2023)

What instructors get

Coarse averages that say how students felt but rarely why, and that are hard to translate into changed practice. (Sarfati, 2022; Spooren et al., 2013)

Limitations of current practice

Three interrelated limitations

1

Quantitative rigidity

Averages and composites are easy to aggregate but compress complex experiences into predetermined categories. Both parties are disadvantaged by the instrument itself.

(Spooren et al., 2013)

2

Poor contextual depth

Likert items quantify but explain little. Written comments carry more insight into teaching and learning, yet standard prompts are general by design.

(Llorent-Bedmar & Cobano-Delgado Palma, 2019; Sun & Yan, 2023)

3

Difficult interpretation

A concern is flagged, but not what to change or how. Low response rates and respondent bias compound the problem.

(Boring et al., 2016; Quansah et al., 2024; Sarfati, 2022)

Background

Two lines of research, rarely combined

Adaptive surveys and dynamic probing

Questions adapt to earlier answers, and probing yields richer data than respondents volunteer on their own (Montgomery & Rossiter, 2022; Robinson, 2023; Tourangeau et al., 2017). LLM-generated follow-ups increase engagement with open-ended questions (Lei et al., 2025; Williams & Ingleby, 2025; Xiao et al., 2020).

AI-assisted qualitative analysis

GPT-4 has coded thousands of open comments with performance comparable to human coders, and can summarize, classify, and surface themes (Liu et al., 2025; Parker et al., 2025; Tai et al., 2024). Human oversight is still required (Shaik et al., 2022; Tai et al., 2024).

The gap

Both show positive results, but little work combines them in the specific context of course evaluation.

This paper

Three contributions

1

Design of Survey DIVA

Dynamic Insight & Validation Agent

A web-based course evaluation instrument with an LLM embedded directly in the evaluation process, together with the ethical safeguards intended to make AI-assisted evaluation trustworthy in practice.

2

Preliminary baseline deployment

Results from a static baseline survey of student attitudes toward the existing SAI process, administered through the platform itself.

3

Design of a controlled evaluation

A section-randomized study of response richness, participation burden, and instructor usefulness of dynamic probing.

Survey DIVA framework

From after the survey to inside it

Current practice

Survey design
↓
Static questions
↓
Responses and data collection
survey closes
GenAI analysis
themes, summaries
↓
Report

Survey DIVA (Dynamic Insight & Validation Agent)

Survey design and static questions (unchanged core)
↓

during the survey

Responses
⇄
GenAI (LLM)
probes responses in real time
⇄
Dynamic follow-up
↓
GenAI analysis
coding, themes, summaries
↓
Report and instructor dashboard

Dynamic probing

Follow-ups come from what the student wrote

Main survey (static)
Dynamic follow-up (LLM-generated)

What is one change that would most improve this course?

“more review content”
→ GenAI →
What type of review content would you find most helpful?

What was the most valuable part of this course for you?

“working on projects with my friends”
→ GenAI →
How did working on projects with friends enhance your learning experience?

How did the course workload feel relative to credit hours?

“it felt overbearing”
→ GenAI →
Can you describe a specific time when the workload felt overbearing?

Implementation

How the probing mechanism works

Pipeline

1

Student completes the static portion of the SAI

2

Platform gathers answers to instructor-designated adaptive text questions (at least 15 characters)

3

One call to the LLM with question text and answers only, never identifiers

4

Model returns a fixed JSON contract listing each proposed follow-up and the response that triggered it

5

Follow-ups are appended as ordinary text questions, capped at two per response

Design choices

Decision rules in the system prompt

Probe vague, unexplained, or unelaborated answers; skip ones that are already detailed. Instructors can override per question.

Fail-safe by default

If the API call fails or returns unparseable output, the survey completes without follow-ups. A model failure can never block submission.

Server-rendered app

Relational database; surveys, responses, and per-question timing paradata stored as structured records; one submission per link.

Analytics dashboard

Self-comparison, not ranking

Course context makes comparison across instructors unfair, so the dashboard is built for an instructor comparing their own results across semesters.

AI summary

High-level summary of student responses for any selected survey, with question-specific summaries on drill-down.

Likert visualizations

Averages plotted alongside previously administered SAIs so instructors can track their own trajectory over time.

Paradata

Time spent per question, presented as a possible indicator of engagement and response behavior.

Raw responses stay one click away from every AI summary, so instructors can always check the model's work.

Ethics and safeguards

Richer feedback means more sensitive data

Concerns 

Bias

Training-data bias could shape which responses get probed and how summaries read.

Transparency

Models can answer without showing the reasoning, which undermines instructor trust (Parker et al., 2025).

Privacy

Open responses go to a third-party LLM provider; neglecting this erodes honest feedback (Li et al., 2025).

Mitigations in Survey DIVA

No personally identifiable information collected; only question text and answers are transmitted

AI as a support tool: raw responses always available alongside AI summaries

Every follow-up is tied to the response that triggered it, so probing is auditable

Informed consent, IRB review, and planned guardrails against probing sensitive topics

Preliminary deployment

Baseline survey of the existing SAI process

309

students invited

38

unique responses (42 submitted)

12%

response rate

What this deployment was

  • Static portion only; dynamic probing was not enabled
  • Instrument: 2 UMUX-Lite usability items (Lewis et al., 2013), 9 adapted Intrinsic Motivation Inventory items (Ryan, 1982), 2 response-habit items, 3 open-ended questions
  • IRB reviewed and determined exempt; voluntary, anonymous, no PII collected
  • Confirmed the platform operates reliably in a live setting

Baseline findings

Usable and important, but not engaging

Measure (5-point scale)Mean (SD)
SAI ease of use (UMUX-Lite)3.89 (1.03)
SAI meets feedback needs (UMUX-Lite)3.66 (0.99)
IMI value / usefulness3.68 (1.29)
IMI effort / importance3.04 (1.06)
IMI interest / enjoyment2.74 (1.14)
Completes all SAIs27 / 38 (71%)
Answers open-ended questions27 / 38 (71%)

82%

agreed the existing SAI is easy to use

66%

agreed it meets their needs for giving feedback

Takeaway

Usability is not the problem. Value scored well above enjoyment: SAIs are seen as important, not engaging.

Open-ended responses

What to keep, and what is missing

Keep

13 of 17

named the open-ended comments as what should remain in future SAIs

Missing

4 of 6

asked for more course-specific or varied questions, including workload items

Also: a wider response range; knowing where the information goes

Top complaint

4 of 8

said fixed questions do not ask what students want to share

Also: end-of-semester timing and fatigue (2), redundant length (2), doubt that feedback leads to change (1)

Students value open-ended expression and want questions that fit their course. That is exactly what dynamic probing is built to provide.

Limitations

What these findings do not show

Small, self-selected sample from a single institution
12% response rate makes nonresponse bias likely
IMI items were adapted for this context and not independently validated
AI-assisted thematic analysis was reviewed by the authors against the raw responses, but not by an independent human coder

Most importantly

Dynamic probing was not enabled, so this study offers no evidence yet about the tool's central mechanism.

That question, along with instructor-facing evaluation, is the subject of the controlled study described next.

Planned controlled evaluation

A section-randomized controlled evaluation

Control sections receive the standard static SAI; treatment sections receive the identical instrument followed by LLM-generated follow-ups.

H1

Richer, more actionable responses

Length, specificity, and actionability coded by human raters blind to condition

H2

No cost to participation

Completion rates not reduced and completion times not meaningfully increased, measured through paradata

H3

More useful, trustworthy summaries

Instructors rate the AI summaries against standard SAI reports in post-semester surveys

Alongside  Account for clustering of students within sections; validate AI-assisted coding against independent human coding with inter-rater agreement; audit generated follow-ups for bias, appropriateness, and safe handling of sensitive topics.

Takeaways

Easy but unengaging, and what to do about it

1

Traditional SAIs constrain students and leave instructors with results they cannot act on. Usability is not the problem; engagement and specificity are.

2

Survey DIVA puts the LLM inside the evaluation: real-time follow-ups, AI-assisted coding and summaries, and a self-comparison dashboard with paradata.

3

The controlled evaluation is next. The goal is to show that generative AI can be integrated into educational assessment responsibly.

Questions?

Jerrick Little, Axel Smart, Lucas Cordova, Heather Kitada Smalley · Willamette University

References · 1 of 3

Boring, A., Ottoboni, K., & Stark, P. B. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1

Hornstein, H. A. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty performance. Cogent Education, 4(1), Article 1304016. https://doi.org/10.1080/2331186X.2017.1304016

Lakeman, R., Coutts, R., Hutchinson, M., Massey, D., Nasrawi, D., Fielden, J., & Lee, M. (2023). Playing the SET game: How teachers view the impact of student evaluation on the experience of teaching and learning. Assessment & Evaluation in Higher Education, 48(6), 749–759. https://doi.org/10.1080/02602938.2022.2126430

Lei, K., Ladenburg, A., Petiwala, Z. K., Wang, Z., Jhawar, D., Bisht, I., Kumar, A., & Lee, D. T. (2025). Dynamic surveys: Using LLMs to blend qualitative depth, quantitative structure, and collaborative interaction. Proceedings of the ACM on Human-Computer Interaction, 9(7), Article CSCW405. https://doi.org/10.1145/3757586

Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013). UMUX-LITE: When there's no time for the SUS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2099–2102). ACM. https://doi.org/10.1145/2470654.2481287

Li, X., Yan, X., & Lai, H. (2025). The ethical challenges in the integration of artificial intelligence and large language models in medical education: A scoping review. PLOS ONE, 20(10), Article e0333411. https://doi.org/10.1371/journal.pone.0333411

Liu, X., Zambrano, A. F., Baker, R. S., Barany, A., Ocumpaugh, J., Zhang, J., Pankiewicz, M., Nasiar, N., & Wei, Z. (2025). Qualitative coding with GPT-4: Where it works better. Journal of Learning Analytics, 12(1), 169–185. https://doi.org/10.18608/jla.2025.8575

Llorent-Bedmar, V., & Cobano-Delgado Palma, V. C. (2019). Análisis crítico de las encuestas universitarias de satisfacción docente [Critical analysis of university teaching satisfaction surveys]. Revista de Educación, 385, 91–117. https://doi.org/10.4438/1988-592X-RE-2019-385-418

References · 2 of 3

Montgomery, J. M., & Rossiter, E. L. (2022). Adaptive inventories: A practical guide for applied researchers. Cambridge University Press. https://doi.org/10.1017/9781108862516

Parker, M. J., Anderson, C., Stone, C., & Oh, Y. (2025). A large language model approach to educational survey feedback analysis. International Journal of Artificial Intelligence in Education, 35(2), 444–481. https://doi.org/10.1007/s40593-024-00414-0

Quansah, F., Cobbinah, A., Asamoah-Gyimah, K., & Hagan, J. E. (2024). Validity of student evaluation of teaching in higher education: A systematic review. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1329734

Robinson, O. C. (2023). Probing in qualitative research interviews: Theory and practice. Qualitative Research in Psychology, 20(3), 382–397. https://doi.org/10.1080/14780887.2023.2238625

Ryan, R. M. (1982). Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology, 43(3), 450–461. https://doi.org/10.1037/0022-3514.43.3.450

Sarfati, G. (2022). We can work it out: Faculty interpretation of student evaluations of teaching. Journal of Further and Higher Education, 46(5), 708–720. https://doi.org/10.1080/0309877X.2021.2002282

Shaik, T., Tao, X., Li, Y., Dann, C., McDonald, J., Redmond, P., & Galligan, L. (2022). A review of the trends and challenges in adopting natural language processing methods for education feedback analysis. IEEE Access, 10. https://doi.org/10.1109/ACCESS.2022.3177752

Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83, 598–642. https://doi.org/10.3102/0034654313496870

References · 3 of 3

Sun, J., & Yan, L. (2023). Using topic modeling to understand comments in student evaluations of teaching. Discover Education, 2(1), Article 25. https://doi.org/10.1007/s44217-023-00051-0

Tai, R. H., Bentley, L. R., Xia, X., Sitt, J. M., Fankhauser, S. C., Chicas-Mosier, A. M., & Monteith, B. G. (2024). An examination of the use of large language models to aid analysis of textual data. International Journal of Qualitative Methods, 23. https://doi.org/10.1177/16094069241231168

Tourangeau, R., Brick, J. M., Lohr, S., & Li, J. (2017). Adaptive and responsive survey designs: A review and assessment. Journal of the Royal Statistical Society: Series A (Statistics in Society), 180(1), 203–223. https://doi.org/10.1111/rssa.12186

Vargas-Madriz, L. F., & Nocente, N. (2023). Exploring students' willingness to provide feedback: A mixed methods research on end-of-term student evaluations of teaching. Social Sciences & Humanities Open, 8(1), Article 100525. https://doi.org/10.1016/j.ssaho.2023.100525

Williams, R. T., & Ingleby, E. (2025). The online survey in qualitative research: Can AI act as a probing tool? Frontiers in Research Metrics and Analytics, 10. https://doi.org/10.3389/frma.2025.1519008

Xiao, Z., Zhou, M. X., Liao, Q. V., Mark, G., Chi, C., Chen, W., & Yang, H. (2020). Tell me about yourself: Using an AI-powered chatbot to conduct conversational surveys with open-ended questions. ACM Transactions on Computer-Human Interaction, 27(3), Article 15. https://doi.org/10.1145/3381804

Yan, L., Sha, L., Zhao, L., Li, Y., Martinez-Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024). Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55(1), 90–112. https://doi.org/10.1111/bjet.13370

Yan, Y., & Liu, H. (2025). Ethical framework for AI education based on large language models. Education and Information Technologies, 30(8), 10891–10909. https://doi.org/10.1007/s10639-024-13241-6

Easy but Unengaging: Rethinking Student Assessments of Instruction with LLM-Driven Dynamic Probing 1

By Lucas Cordova

Easy but Unengaging: Rethinking Student Assessments of Instruction with LLM-Driven Dynamic Probing 1

CCSC Northwest 2026 paper presentation

  • 12

More from Lucas Cordova