Reply to Ledgerwood: Predictions without analysis plans are inert

Abstract

Ledgerwood (1) argues that there are two independent uses of preregistration that are conflated in Nosek et al. (2) and elsewhere: “Preregistering theoretical predictions enables theory falsifiability. Preregistering analysis plans enables type I error control.” We appreciate that the comment elevates the complementary roles of prediction and analysis plans in preregistration. We disagree that they are conflated in the sense of being “two types of preregistration.”

To enable theory falsification, we agree that a preregistration should offer a prediction derived from theory and provide the theoretical context. However, a prediction without an analysis plan is inert for falsification. An analysis plan is necessary to specify how the prediction will be tested with the observed data. So, the position that prediction and analysis plans are conflated is misleading—theory testing requires both.

Here is a somewhat different way to describe the distinct roles of prediction and analysis plans in preregistration to elaborate on Ledgerwood’s (1) points. Researcher 1 preregisters that they will conduct a two-tailed t test with alpha = 0.05 to evaluate whether subjects randomly assigned to condition A versus condition B will differ on a single outcome. Researcher 1 does not preregister a directional prediction. Researcher 2 observes the study design and analysis plan and preregisters a prediction of A > B based on their theoretical framework. Researcher 3 does the same and predicts B > A. The observed outcome is P = 0.01 for B > A, opposite of researcher 2’s prediction and consistent with researcher 3’s.

Is this an exploratory result because researcher 1 did not make a directional prediction? Is it a theoretical falsification because researcher 2’s prediction was not supported? Is it a theoretical confirmation because researcher 3’s prediction was supporting? “Yes” and “no” are defensible assertions for all three questions.

The statistical outcomes do not know what the researchers predicted. The P value is interpretable the same way for all researchers (Ledgerwood’s error control). This might imply that all three researchers should now believe that B > A to the same degree based on the statistical evidence. Not so. Predictions are an informal way that prior beliefs are incorporated into null hypothesis significance testing. Falsification is rarely a discrete event, nor is it a consensus event. All three researchers should be responsive to the new evidence, but the preexisting beliefs supporting their predictions will shape how much their beliefs change with the evidence. This updating is not quantified directly by the P value.

Preregistering predictions is also useful for the social aspect of scientific communication. In basketball, hitting a bank shot is much more impressive if it is called before shooting. Otherwise, people presume that it was luck. Researcher 3’s theoretical perspective for B > A may gain greater credibility because of its successful prediction compared with generating an explanation after the fact. This may invite greater empirical scrutiny to assess whether the theoretical perspective survives more prediction scenarios.

Echoing Ledgerwood (1), preregistration without predictions is fine for exploration. However, all preregistered predictions need analysis plans specifying how the prediction will be evaluated when the outcomes are observed.