University of Sussex
Browse

Many analysts, one data set: making transparent how variations in analytic choices affect results

Download (246.69 kB)
Version 2 2023-06-12, 07:25
Version 1 2023-06-09, 14:47
journal contribution
posted on 2023-06-12, 07:25 authored by Raphael Silberzahn, E L Uhlmann, D P Martin, P Anselmi, F Aust, E Awtrey, Š Bahník, F Bai, C Bannard, E Bonnier, R Carlsson, F Cheung, G Christensen, R. Clay, M A Craig, A D Rosa, L Dam, M H Evans, I Flores Cervantes, N Fong, M Gamez-Djokic, A Glenz, S Gordon-McKeon, T J HEaton Heaton, K Hederos, M Heene, A J Hofelich Mohr, F Högden, K Hui, M Johannesson, J Kalodimos, E Kaszubowski, D M Kennedy, R Lei, T A Lindsay, S Liverani, C R Madan, D Molden, E Molleman, R D Morey, L B Mulder, B.R Nijstad, N G Pope, B Pope, J M Prenoveau, F Rink, E Robusto, H Roderique, A Sandberg, E Schlüter, F D Schönbrodt, M F Sherman, S A Sommer, K Sotak, S Spain, C Spörlein, T Stafford, L Stefanutti, S Tauber, J Ullrich, M Vianello, E-J Wagenmakers, M Witkowiak, S Yoon, B A Nosek
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

History

Publication status

  • Published

File Version

  • Published version

Journal

Advances in Methods and Practices in Psychological Science

ISSN

2515-2459

Publisher

SAGE Publications

Issue

3

Volume

1

Page range

337-355

Department affiliated with

  • Management Publications

Full text available

  • Yes

Peer reviewed?

  • Yes

Legacy Posted Date

2018-08-28

First Open Access (FOA) Date

2018-08-28

First Compliant Deposit (FCD) Date

2018-08-28

Usage metrics

    University of Sussex (Publications)

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC