Not Everyone Writes Good Examples
But Good Examples Can Come From Anywhere
HCOMP 2019
Shayan Doroudi, Ece Kamar, and Emma Brunskill
Can we use these to help other learners learn?
Learners Naturally Generate Artifacts
Peer review
Workers Naturally Generate Artifacts
Can we use these to help other workers learn?
Want workers to become more skilled
No existing curriculum
Tasks changing over time
Two Ways to View This Talk
Training crowdworkers to do complex tasks
Exploring the efficacy of learner-generated work in teaching other learners
Learning for Crowdworkers
Crowdsourcing for Learners
Learnersourced Content Creation
-
Learnersourcing: “Crowdsourcing with learners as the crowd”
-
Using work done by crowds of learners to help improve the educational experience of future learners [Kim et al., 2015].
-
- Prior work has shown crowdsourcing can be used to create effective content---in some cases on par with expert content [Aleahmad et al. 2009; Williams et al., 2016; Whitehill & Seltzer, 2017].
- However, we are interested in passive learnersourcing:
using artifacts that learners naturally generate.
Product Comparison Task
Example Collection
56
examples collected
47
examples collected
Worker Examples
Experiment 1
Randomly Selected
Experiment 1: Results
No sig. differences between conditions
Experiment 1: Results
No sig. differences between conditions
Experiment 1: Results
No sig. differences between conditions
Experiment 2
3 Highest Quality Examples
Experiment 2: Results
Sig. difference between two good examples and control on Task A'
(Mann-Whitney U test, \(p < 0.01\))
Experiment 2: Results
Sig. difference between two good examples and control on Task A'
(Mann-Whitney U test, \(p < 0.01\))
Experiment 2: Results
Sig. difference between two good examples and control on Task A'
(Mann-Whitney U test, \(p < 0.01\))
Top Three Examples A
Top Three Examples B
Is Quality all that Matters?
Conclusion
Future direction: use machine learning models to curate better solutions
Our results suggest other features beyond quality may help identify more pedagogically effective examples.
and to learn more about how peer work leads to learning.
Worker's work can be effectively presented as examples to future works, provided the work is sufficiently high quality
Learnersourced Content as a Tool for Scientific Discovery
Peer Work
ML
Model
Can lead to new insights about how people learn.
Length
Spacing
Word choice
What makes an example
pedagogically effective?
Quality
for each learner/worker
Acknowledgements
In the past, I have made no secret of my disdain for
Chef Gusteau’s famous motto: ‘Anyone can cook.’ But I realize,
only now do I truly understand what he meant. Not everyone can
become a great artist, but a great artist can come from anywhere.
- Anton Ego, Ratatouille
This research was supported in part by a Google grant and a Microsoft Research Faculty Fellowship.
Product Comparison Task Rubric
5. It's hard to imagine a more useful resource for someone to decide which product to buy. The review appears to contain no factual errors.
4. The review would help you decide which product is best, but could have had some more information or could have been structured better.
3. The review would be helpful, but you would need to do more research to decide which product to buy.
2. The review has some distinctions between the two products, but you basically need to do your own research from scratch to decide which product to buy.
1. The review is misleading or does not really contain useful information (e.g., contains a major factual error that could result in purchasing the wrong product).