"Simple things that don't work"

Correlating peer review grades via scoring rules

Correlating peer reviewer variance across assignments

Inferring skill on simulated data

Questions

• Using a higher-dimensional algorithm
  • Is the submission error consistent across rubric dimensions, or are there groupings?
• Incorporate groundtruth scores in expectation without affecting the calculation of peer variance