With Low Power Comes Low Credibility? Toward a Principled Critique of Results From Underpowered Tests

Abstract

Researchers should be motivated to adequately power statistical tests because tests with low power have a low probability of detecting true effects. However, it is also often claimed that significant results obtained by underpowered tests are less likely to reflect a true effect. Here, we critically discuss this “low-power/low-credibility” (LPLC) critique from both frequentist and Bayesian perspectives. Although the LPLC critique is first and foremost a critique of frequentist tests, it is itself not consistent with frequentist theory. In particular, it demands that researchers have some information on the probability that a hypothesis is true before they test it. However, such prior probabilities are dismissed as meaningless in frequentist inference, and we demonstrate that they cannot be meaningfully introduced into frequentist thinking. Even when adopting a Bayesian perspective, however, significant results from tests with low power can provide a nonnegligible amount of support for the tested hypothesis. We conclude that even though low power reduces the chances to obtain significant findings, there is little justification to dismiss already obtained significant findings on the basis of low power alone. However, concerns about low power might often reflect suspicions that findings were produced by questionable research practices. If this is the case, we suggest that communicating these issues transparently rather than using low power as a proxy concern will be more appropriate. We conclude by providing suggestions on how results from tests with low power can be critiqued for the correct reasons and in a constructive manner.