Vít Gabrhel
vit.gabrhel@mail.muni.cz
vit.gabrhel@cdv.cz
FSS MU,
7. 11. 2017
1. Úvod
2. Meta-analýza
3. Reproducibility, replicability etc.
"No isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon."
Fisher, 1971, s. 13
Open Science, 2015
"We conducted a large-scale, collaborative effort to obtain an initial estimate of the reproducibility of psychological science"
Původní
Průměrná velikost účinku:
Mr = 0,403; SD = 0,188
Statistická signifikance:
97 % studií p < 0,05
Replikace
Průměrná velikost účinku:
Mr = 0,197; SD= 0,257
Statistická signifikance:
36 % studií p < 0,05
100 studií a výsledky jejich replikace
Hodnota velikostí účinku z původních studií se nacházela v 95% intervalu spolehlivosti při replikaci v 47 % případů
"Věda má kumulativní povahu, ke studiím však přistupujeme nikoli jako jedné z mnoha, ale izolovaně, stojícími o sobě."
Chalmers, cit. dle Borenstein et al., 2009
Narativní review
Od 90. let 20. st. přechod k Metaanalýze a Systematic Review
Metaanalýza / Systematic review
Systematic review
Meta-analýza
Velikost účinku (Size effect)
Váhy
Přesnost (Precision)
Hodnota p
Meta-analýza je kvalitní do té míry, do jaké jsou kvalitní (nezkreslené apod.) individuální studie
Jedno číslo nemůže zastupovat celou výzkumnou oblast
Studie
Úroveň prováděných meta-analýz není dobrá
Silná orientace na statisticky signifikantní výsledek
Bakker, Van Dijk, & Wicherts, 2012
Fraley & Marks, 2007
Jenže...
"Consequently, if all effects reported in published studies were true, only 35% would be replicable in similarly underpowered studies." Asendorpf et al. 2013, s. 110
In a poll of more than 2000 psychologists, prevalences of ‘Deciding whether to collect more data after looking to see whether the results were significant’ and ‘Stopping data collection earlier than planned because one found the result that one had been looking for’ were subjectively estimated at 61% and 39%, respectively.
John, Loewenstein, & Prelec, cit. dle Asendorpf et al., 2013
Podvodné versus pochybné jednání
Fraud is typically limited to cases in which researchers create false data.
In contrast, QRPs typically involve the exclusion of data that are inconsistent with a theoretical hypothesis. QRPs are treated differently than fraud because QRPs can sometimes be used for legitimate purposes.
John, Loewenstein, & Prelec, 2012
Reproducibility (Reprodukovatelnost)
"Researcher B must have the following: (a) the raw data; (b) the code book (variable names and labels, value labels, and codes formissing data); and (c) knowledge of the analyses that were performed by Researcher A (e.g. the syntax of a statistics program)."
Asendorpf et al., 2013, s. 109
Replicability (Replikovatelnost)
"The finding can be obtained with other random samples drawn from a multidimensional space that captures the most important facets of the research design. In psychology, the facets typically include the following: (a) individuals (or dyads or groups); (b) situations (natural or experimental); (c) operationalizations (experimental manipu- lations, methods, and measures); and (d) time points."
Asendorpf et al., 2013, s. 109
Generalizability (Zobecnitelnost)
"It does not depend on an originally unmeasured variable that has a systematic effect. In psychology, generalizability is often demonstrated by showing that a potential moderator variable has no effect on a group difference or correlation."
Asendorpf et al., 2013, s. 110
Zmenšit chybu měření
Od "p < 0,05" k
Autoři studií - důraz na větší míru transparence
Žurnály, recenzenti, editoři
Aneb co mají studenti chtít po svých učitelích?
Podpora
Aneb co mají studenti chtít po svých učitelích?
Změna Publish or Perish politiky
Alternativa
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, ... & Wicherts, J. M. (2013). Recommendations for Increasing Replicability in Psychology. European Journal of Personality, 27(2), 108–119. doi: 10.1002/per.1919
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. Chichester, UK: John Wiley & Sons.
Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS One, 5, e10068.
Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.
Fisher, R. (1971). The Design of Experiments. New York: Hafner Publishing Company.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532.
Open Science (2015). Estimating the reproducibility of psychological science. 349 (6251), http://doi.org/10.1126/science.aac4716
Taylor, L. E., Swerdfeger, A. L., & Eslick, G. D. (2014). Vaccines are not associated with autism: an evidence-based meta-analysis of case-control and cohort studies. Vaccine, 32(29), 3623–3629. http://doi.org/10.1016/j.vaccine.2014.04.085