Replicability

An important part of writing a paper is polishing the paper. You start with a first draft but then you find small mistakes, things to add or to remove. Which leads to redoing the analysis and a second, third and fourth draft of the paper. And then you get comments at seminars and from referees, further increasing the number of re-analyses and re-writes.

[*AoI = “Articles of Interest” is a feature of TRN where we report abstracts of recent research related to replication and research integrity.] ABSTRACT (taken from the article) “This article reviews and summarizes current reproduction and replication practices in political science. We first provide definitions for reproducibility and replicability. We then review data availability policies for 28 leading political science journals and present the results from a survey of editors about their willingness to publish comments and replications.

“Replicability of findings is at the heart of any empirical science” (Asendorpf, Conner, De Fruyt, et al., 2013, p. 108) The idea that scientific results should be reliably demonstrable under controlled circumstances has a special status in science. In contrast to our high expectations for replicability, unfortunately, recent reports suggest that only about 36% (Open Science Collaboration, 2015) to 62% (Camerer, Dreber, Holzmeister, et al.

*[This blog draws on the article “*The statistical significance filter leads to overoptimistic expectations of replicability”, authored by Shravan Vasishth, Daniela Mertzen, Lena A. Jäger, and Andrew Gelman, published in the Journal of Memory and Language, 103, 151-175, 2018. An open access version of the article is available here.] The Problem Statistics textbooks tell us that the sample mean is an unbiased estimate of the true mean.

[This blog is a summary of a longer treatment of the subject that was published in Frontiers in Psychology in June 2017. To read that article, click here.] Physicists have asked “why is there something rather than nothing?” They have theorized that it had to do with the formation of an asymmetry between matter and antimatter in the fractions of milliseconds after the Big Bang.

Statisticians have been keen to critique statistical aspects of the “replication crisis” in other scientific disciplines. But new statistical tools are often published and promoted without any thought to replicability. This needs to change, argue …

We examine 244 independent tests of interaction effects published in recent issues of four leading journals in the organizational sciences in order to estimate the replicability of reported statistically significant interaction effects. A z-curve …

The credibility revolution in social science has led to the recommendation and adoption of practices to increase the replicability of scientific findings. Many of the recommended practices, such as replication and preregistration, present unique …

Ecological momentary assessment (EMA) represents a promising approach to study cognitive aging. In contrast to laboratory-based studies, EMA involves the repeated sampling of experiences in daily life contexts, enabling investigators to gain access …

There has been much debate in recent years about how open research practices, which have been promoted in efforts to improve research robustness, may (not) be appropriate for qualitative methodologies, particularly in educational research. Among …