Replication and the Manufacture of Scientific Inferences: A Formal Approach

Abstract

The field of replication studies remains a controversial, misunderstood, and unappreciated piñata of 18 replication typologies spanning 79 replication types. To help bring order to the chaos, I contribute a theory of manufactured inferences. The theory is built on three pillars: (1) replication causal diagrams (or r-dags for short), (2) a formal conceptualization of study procedures, and (3) the use of Bayesian inference to update our beliefs about the natural phenomenon under investigation and the operating characteristics of the study procedures used to study it. I use this theory to motivate a formal typology of replication types, explaining how they are done and for what purpose. Finally, I discuss some implications of this theory, including the importance of an analytical approach to robustness and generalizability replications, the need to avoid conceptual replications, the possibility of legitimate (unplanned) specification searches, the limitations of meta-analysis, and the false dichotomy between so-called successful and failed replications.