The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science: Comments on Van Drimmelen (2023)

Abstract

I discuss Van Drimmelen’s (2023) Metascience2023 presentation on researchers’ decision making during the research process. In particular, I consider his evidence that researchers’ discretion over research decisions is unavoidable when they follow research plans that are either overdetermined (i.e., too prescriptive) or underdetermined (i.e., too vague). I argue that this evidence points to a prescriptiveness trade-off when writing preregistered plans: All other things being equal, plans that are more prescriptive are more likely to result in deviations that turn their confirmatory tests into exploratory tests, and plans that are less prescriptive are more likely to result in confirmatory tests that are susceptible to questionable research practices. I also consider Van Drimmelen’s idea that researchers may make unconscious, implicit decisions during the research process. I relate these implicit decisions to Rumsfeld’s (2002) concept of unknown unknowns: “the things we don’t know we don’t know”! I argue that scientists can report their known knowns (what they know they did and found), and they can be transparent and speculative about their known unknowns (what they know they didn’t do and may find), but that they can’t say much about their unknown unknowns (including their unconscious, implicit decisions) because, by definition, they don’t know what they are! Nonetheless, I think that it’s important to acknowledge unknown unknowns in science because doing so helps to contextualise research efforts as being highly tentative and fallible.