The preregistration revolution needs to distinguish between predictions and analyses

Abstract

Nosek et al. (1) recently joined others in advocating for “widespread adoption of preregistration” as a tool for advancing science. The language they use in making this important argument, however, creates unnecessary confusion: Like many others discussing these issues, they seem to conflate the goal of theory falsification with the goal of constraining type I error. This masks a crucial distinction between two types of preregistration: preregistering a theoretical, a priori, directional prediction (which serves to clarify how a hypothesis is constructed) and preregistering an analysis plan (which serves to clarify how evidence is produced).

Indeed, philosophers of science have identified elements of both how a hypothesis is constructed and how evidence is produced that are important for scientifically valid inference (2–4). We can distill these to two key, separable questions:

i) Have these data influenced my theoretical prediction? This question is relevant when researchers want to test existing theory: Rationally speaking, we should only adjust our confidence in a theory in response to evidence that was not itself used to construct the theoretical prediction in question (3). Preregistering theoretical predictions can help researchers distinguish clearly between using evidence to inform versus test theory (3, 5, 6).

ii) Have these data influenced my choice of statistical test (and/or other dataset-construction/analysis decisions)? This question is relevant when researchers want to know the type I error rate of statistical tests: Flexibility in researcher decisions can inflate the risk of false positives (7, 8). Preregistration of analysis plans can help researchers distinguish clearly between data-dependent analyses (which can be interesting but may have unknown type I error) and data-independent analyses (for which P values can be interpreted as diagnostic about the likelihood of a result; refs. 1 and 9).

Put differently, preregistration of theoretical predictions helps researchers know how to correctly calibrate their confidence that a study tests (versus informs) a theory, whereas preregistration of analysis plans helps researchers know how to correctly calibrate their confidence that a specific finding is unlikely to be due to chance.

Conflating theoretical predictions and analyses is problematic for multiple reasons. First, it implies, erroneously, that preanalysis plans can only help control type I error when research is in a prediction-making/theory-testing phase (e.g., theory Z predicts a gender difference in trait X; ref. 1). In fact, preanalysis plans can also be useful in the question-asking/discovery/theory-building phase (e.g., is there a gender difference in trait X?). Second, it may lead people to preregister the wrong things (e.g., a researcher attempting to control type I error records careful predictions but omits or only loosely specifies a preanalysis plan). Third, it increases misunderstandings and backlash against preregistration as scientists discuss these issues in everyday life (e.g., students erroneously infer that their results are more robust if they correctly guess them ahead of time; skeptics understandably argue that recording one’s prediction ahead of time has no effect on type I error).

If we want clear communication, productive debates, and effective strategies for advancing science, we must first pull apart our tangled terminology. Preregistering theoretical predictions enables theory falsifiability. Preregistering analysis plans enables type I error control.