Sheng Long
12/03/24
🧭 I had three guiding questions:
Winkler, Robert L., et al. "Scoring rules and the evaluation of probabilities." Test 5 (1996): 1-60.
... for probability assessment
Winkler, Robert L., et al. "Scoring rules and the evaluation of probabilities." Test 5 (1996): 1-60.
Camerer, Colin F., and Robin M. Hogarth. "The effects of financial incentives in experiments: A review and capital-labor-production framework." Journal of risk and uncertainty 19 (1999): 7-42.
cognitive capital
(e.g. heuristics, perceptual skills)
some objective function to maximize
(e.g. maximize monetary payoff while minimizing effort)
Camerer, Colin F., and Robin M. Hogarth. "The effects of financial incentives in experiments: A review and capital-labor-production framework." Journal of risk and uncertainty 19 (1999): 7-42.
Camerer, Colin F., and Robin M. Hogarth. "The effects of financial incentives in experiments: A review and capital-labor-production framework." Journal of risk and uncertainty 19 (1999): 7-42.
Camerer, Colin F., and Robin M. Hogarth. "The effects of financial incentives in experiments: A review and capital-labor-production framework." Journal of risk and uncertainty 19 (1999): 7-42.
Camerer, Colin F., and Robin M. Hogarth. "The effects of financial incentives in experiments: A review and capital-labor-production framework." Journal of risk and uncertainty 19 (1999): 7-42.
Camerer, Colin F., and Robin M. Hogarth. "The effects of financial incentives in experiments: A review and capital-labor-production framework." Journal of risk and uncertainty 19 (1999): 7-42.
Calibration of probabilities: The state of the art to 1980
1982
... meterologists have been interested in calibration as far back as 1906
Calibration of probabilities: The state of the art to 1980
1982
1906
meteorologists
1950
Glenn Brier (meteorologist) proposed an approach to verify the accuracy of probabilistic forecasts
Calibration of probabilities: The state of the art to 1980
1982
1906
meteorologists
1950
Glenn Brier & Brier score
1991
Calibration and probability judgements: Conceptual and methodological issues
... much of the research on calibration is "dust-bowl empiricism"
Calibration of probabilities: The state of the art to 1980
1982
1906
meteorologists
1950
Glenn Brier & Brier score
1991
Calibration and probability judgements: Conceptual and methodological issues
and many many more papers and stylized facts
Encoding Subjective Probabilities: A Psychological and Psychometric Review
subjective probability
1983
2004
Perspectives on probability judgment calibration
1973
Subjective probability and its measurement
e.g. hard-easy effect --- people tend to be over-confident in hard questions but under-confident in easy questions
philosophical tangent : subjective probability vs observed frequency
more related to discrimination
i.e., the ability to distinguish between different outcomes
predictions agreeing with actual rates of outcomes
Lindhiem, Oliver, et al. "The importance of calibration in clinical psychology." Assessment 27.4 (2020): 840-854.
predicted positive | predicted negative | |
---|---|---|
Positive | True positive | False negative |
Negative | False positive | True negative |
confusion/error matrix
Wixted, John T. “The Forgotten History of Signal Detection Theory.” Journal of Experimental Psychology. Learning, Memory, and Cognition, vol. 46, no. 2, Feb. 2020, pp. 201–33. PubMed, https://doi.org/10.1037/xlm0000732.
False positive
True positive
45 degree line \(\implies\) perfect calibration
🤔 How calibrated is this classifier?
Over-extremity
predictions are consistently too close to 0 or 1
To ground things more concretely, suppose we built classifiers for children ADHD detection.
Over-prediction
predicted probability consistently > observed proportion
🤔 How calibrated is this classifier?
Benefits of graphical approaches:
❓Which model(s) has good discrimination and which has good calibration?
A: good discrimination, good calibration
B: good discrimination, bad calibration
C: bad discrimination, good calibration
D: bad discrimination, bad calibration