CCSC Northwest 2026
Rethinking Student Assessments of Instruction with LLM-Driven Dynamic Probing
Jerrick Little, Axel Smart, Lucas Cordova, Heather Kitada Smalley
Willamette University · School of Computing & Information Sciences
The problem
Student Assessments of Instruction feed tenure and promotion decisions, yet most still pair a fixed set of Likert items with one or two open prompts. (Hornstein, 2017; Lakeman et al., 2023; Spooren et al., 2013)
What students get
A static form delivered during finals week, with little sign that their feedback produces change. (Vargas-Madriz & Nocente, 2023)
What instructors get
Coarse averages that say how students felt but rarely why, and that are hard to translate into changed practice. (Sarfati, 2022; Spooren et al., 2013)
Limitations of current practice
1
Quantitative rigidity
Averages and composites are easy to aggregate but compress complex experiences into predetermined categories. Both parties are disadvantaged by the instrument itself.
(Spooren et al., 2013)
2
Poor contextual depth
Likert items quantify but explain little. Written comments carry more insight into teaching and learning, yet standard prompts are general by design.
(Llorent-Bedmar & Cobano-Delgado Palma, 2019; Sun & Yan, 2023)
3
Difficult interpretation
A concern is flagged, but not what to change or how. Low response rates and respondent bias compound the problem.
(Boring et al., 2016; Quansah et al., 2024; Sarfati, 2022)
Background
Adaptive surveys and dynamic probing
Questions adapt to earlier answers, and probing yields richer data than respondents volunteer on their own (Montgomery & Rossiter, 2022; Robinson, 2023; Tourangeau et al., 2017). LLM-generated follow-ups increase engagement with open-ended questions (Lei et al., 2025; Williams & Ingleby, 2025; Xiao et al., 2020).
AI-assisted qualitative analysis
GPT-4 has coded thousands of open comments with performance comparable to human coders, and can summarize, classify, and surface themes (Liu et al., 2025; Parker et al., 2025; Tai et al., 2024). Human oversight is still required (Shaik et al., 2022; Tai et al., 2024).
The gap
Both show positive results, but little work combines them in the specific context of course evaluation.
This paper
1
Design of Survey DIVA
Dynamic Insight & Validation Agent
A web-based course evaluation instrument with an LLM embedded directly in the evaluation process, together with the ethical safeguards intended to make AI-assisted evaluation trustworthy in practice.
2
Preliminary baseline deployment
Results from a static baseline survey of student attitudes toward the existing SAI process, administered through the platform itself.
3
Design of a controlled evaluation
A section-randomized study of response richness, participation burden, and instructor usefulness of dynamic probing.
Survey DIVA framework
Current practice
Survey DIVA (Dynamic Insight & Validation Agent)
during the survey
Dynamic probing
What is one change that would most improve this course?
“more review content”What was the most valuable part of this course for you?
“working on projects with my friends”How did the course workload feel relative to credit hours?
“it felt overbearing”Implementation
Pipeline
Student completes the static portion of the SAI
Platform gathers answers to instructor-designated adaptive text questions (at least 15 characters)
One call to the LLM with question text and answers only, never identifiers
Model returns a fixed JSON contract listing each proposed follow-up and the response that triggered it
Follow-ups are appended as ordinary text questions, capped at two per response
Design choices
Decision rules in the system prompt
Probe vague, unexplained, or unelaborated answers; skip ones that are already detailed. Instructors can override per question.
Fail-safe by default
If the API call fails or returns unparseable output, the survey completes without follow-ups. A model failure can never block submission.
Server-rendered app
Relational database; surveys, responses, and per-question timing paradata stored as structured records; one submission per link.
Analytics dashboard
Course context makes comparison across instructors unfair, so the dashboard is built for an instructor comparing their own results across semesters.
AI summary
High-level summary of student responses for any selected survey, with question-specific summaries on drill-down.
Likert visualizations
Averages plotted alongside previously administered SAIs so instructors can track their own trajectory over time.
Paradata
Time spent per question, presented as a possible indicator of engagement and response behavior.
Raw responses stay one click away from every AI summary, so instructors can always check the model's work.
Ethics and safeguards
Concerns
Bias
Training-data bias could shape which responses get probed and how summaries read.
Transparency
Models can answer without showing the reasoning, which undermines instructor trust (Parker et al., 2025).
Privacy
Open responses go to a third-party LLM provider; neglecting this erodes honest feedback (Li et al., 2025).
Mitigations in Survey DIVA
No personally identifiable information collected; only question text and answers are transmitted
AI as a support tool: raw responses always available alongside AI summaries
Every follow-up is tied to the response that triggered it, so probing is auditable
Informed consent, IRB review, and planned guardrails against probing sensitive topics
Preliminary deployment
309
students invited
38
unique responses (42 submitted)
12%
response rate
What this deployment was
Baseline findings
82%
agreed the existing SAI is easy to use
66%
agreed it meets their needs for giving feedback
Takeaway
Usability is not the problem. Value scored well above enjoyment: SAIs are seen as important, not engaging.
Open-ended responses
Keep
13 of 17
named the open-ended comments as what should remain in future SAIs
Missing
4 of 6
asked for more course-specific or varied questions, including workload items
Also: a wider response range; knowing where the information goes
Top complaint
4 of 8
said fixed questions do not ask what students want to share
Also: end-of-semester timing and fatigue (2), redundant length (2), doubt that feedback leads to change (1)
Students value open-ended expression and want questions that fit their course. That is exactly what dynamic probing is built to provide.
Limitations
Most importantly
Dynamic probing was not enabled, so this study offers no evidence yet about the tool's central mechanism.
That question, along with instructor-facing evaluation, is the subject of the controlled study described next.
Planned controlled evaluation
Control sections receive the standard static SAI; treatment sections receive the identical instrument followed by LLM-generated follow-ups.
H1
Richer, more actionable responses
Length, specificity, and actionability coded by human raters blind to condition
H2
No cost to participation
Completion rates not reduced and completion times not meaningfully increased, measured through paradata
H3
More useful, trustworthy summaries
Instructors rate the AI summaries against standard SAI reports in post-semester surveys
Alongside Account for clustering of students within sections; validate AI-assisted coding against independent human coding with inter-rater agreement; audit generated follow-ups for bias, appropriateness, and safe handling of sensitive topics.
Takeaways
Traditional SAIs constrain students and leave instructors with results they cannot act on. Usability is not the problem; engagement and specificity are.
Survey DIVA puts the LLM inside the evaluation: real-time follow-ups, AI-assisted coding and summaries, and a self-comparison dashboard with paradata.
The controlled evaluation is next. The goal is to show that generative AI can be integrated into educational assessment responsibly.
Questions?
Jerrick Little, Axel Smart, Lucas Cordova, Heather Kitada Smalley · Willamette University
References · 1 of 3
Boring, A., Ottoboni, K., & Stark, P. B. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1
Hornstein, H. A. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty performance. Cogent Education, 4(1), Article 1304016. https://doi.org/10.1080/2331186X.2017.1304016
Lakeman, R., Coutts, R., Hutchinson, M., Massey, D., Nasrawi, D., Fielden, J., & Lee, M. (2023). Playing the SET game: How teachers view the impact of student evaluation on the experience of teaching and learning. Assessment & Evaluation in Higher Education, 48(6), 749–759. https://doi.org/10.1080/02602938.2022.2126430
Lei, K., Ladenburg, A., Petiwala, Z. K., Wang, Z., Jhawar, D., Bisht, I., Kumar, A., & Lee, D. T. (2025). Dynamic surveys: Using LLMs to blend qualitative depth, quantitative structure, and collaborative interaction. Proceedings of the ACM on Human-Computer Interaction, 9(7), Article CSCW405. https://doi.org/10.1145/3757586
Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013). UMUX-LITE: When there's no time for the SUS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2099–2102). ACM. https://doi.org/10.1145/2470654.2481287
Li, X., Yan, X., & Lai, H. (2025). The ethical challenges in the integration of artificial intelligence and large language models in medical education: A scoping review. PLOS ONE, 20(10), Article e0333411. https://doi.org/10.1371/journal.pone.0333411
Liu, X., Zambrano, A. F., Baker, R. S., Barany, A., Ocumpaugh, J., Zhang, J., Pankiewicz, M., Nasiar, N., & Wei, Z. (2025). Qualitative coding with GPT-4: Where it works better. Journal of Learning Analytics, 12(1), 169–185. https://doi.org/10.18608/jla.2025.8575
Llorent-Bedmar, V., & Cobano-Delgado Palma, V. C. (2019). Análisis crítico de las encuestas universitarias de satisfacción docente [Critical analysis of university teaching satisfaction surveys]. Revista de Educación, 385, 91–117. https://doi.org/10.4438/1988-592X-RE-2019-385-418
References · 2 of 3
Montgomery, J. M., & Rossiter, E. L. (2022). Adaptive inventories: A practical guide for applied researchers. Cambridge University Press. https://doi.org/10.1017/9781108862516
Parker, M. J., Anderson, C., Stone, C., & Oh, Y. (2025). A large language model approach to educational survey feedback analysis. International Journal of Artificial Intelligence in Education, 35(2), 444–481. https://doi.org/10.1007/s40593-024-00414-0
Quansah, F., Cobbinah, A., Asamoah-Gyimah, K., & Hagan, J. E. (2024). Validity of student evaluation of teaching in higher education: A systematic review. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1329734
Robinson, O. C. (2023). Probing in qualitative research interviews: Theory and practice. Qualitative Research in Psychology, 20(3), 382–397. https://doi.org/10.1080/14780887.2023.2238625
Ryan, R. M. (1982). Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology, 43(3), 450–461. https://doi.org/10.1037/0022-3514.43.3.450
Sarfati, G. (2022). We can work it out: Faculty interpretation of student evaluations of teaching. Journal of Further and Higher Education, 46(5), 708–720. https://doi.org/10.1080/0309877X.2021.2002282
Shaik, T., Tao, X., Li, Y., Dann, C., McDonald, J., Redmond, P., & Galligan, L. (2022). A review of the trends and challenges in adopting natural language processing methods for education feedback analysis. IEEE Access, 10. https://doi.org/10.1109/ACCESS.2022.3177752
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83, 598–642. https://doi.org/10.3102/0034654313496870
References · 3 of 3
Sun, J., & Yan, L. (2023). Using topic modeling to understand comments in student evaluations of teaching. Discover Education, 2(1), Article 25. https://doi.org/10.1007/s44217-023-00051-0
Tai, R. H., Bentley, L. R., Xia, X., Sitt, J. M., Fankhauser, S. C., Chicas-Mosier, A. M., & Monteith, B. G. (2024). An examination of the use of large language models to aid analysis of textual data. International Journal of Qualitative Methods, 23. https://doi.org/10.1177/16094069241231168
Tourangeau, R., Brick, J. M., Lohr, S., & Li, J. (2017). Adaptive and responsive survey designs: A review and assessment. Journal of the Royal Statistical Society: Series A (Statistics in Society), 180(1), 203–223. https://doi.org/10.1111/rssa.12186
Vargas-Madriz, L. F., & Nocente, N. (2023). Exploring students' willingness to provide feedback: A mixed methods research on end-of-term student evaluations of teaching. Social Sciences & Humanities Open, 8(1), Article 100525. https://doi.org/10.1016/j.ssaho.2023.100525
Williams, R. T., & Ingleby, E. (2025). The online survey in qualitative research: Can AI act as a probing tool? Frontiers in Research Metrics and Analytics, 10. https://doi.org/10.3389/frma.2025.1519008
Xiao, Z., Zhou, M. X., Liao, Q. V., Mark, G., Chi, C., Chen, W., & Yang, H. (2020). Tell me about yourself: Using an AI-powered chatbot to conduct conversational surveys with open-ended questions. ACM Transactions on Computer-Human Interaction, 27(3), Article 15. https://doi.org/10.1145/3381804
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez-Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024). Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55(1), 90–112. https://doi.org/10.1111/bjet.13370
Yan, Y., & Liu, H. (2025). Ethical framework for AI education based on large language models. Education and Information Technologies, 30(8), 10891–10909. https://doi.org/10.1007/s10639-024-13241-6