On the Surprising Longevity of Flogged Horses: Why There Is a Case for the Significance Test

Abstract

Criticisms of null-hypothesis significance tests (NHSTs) are reviewed. Used as formal, two-valued decision procedures, they often generate misleading conclusions. However, critics who argue that NHSTs are totally meaningless because the null hypothesis is virtually always false are overstating their case. Critics also neglect the whole class of valuable significance tests that assess goodness of fit of models to data. Even as applied to simple mean differences, NHSTs can be rhetorically useful in defending research against criticisms that random factors adequately explain the results, or that the direction of mean difference was not demonstrated convincingly. Principled argument and counterargument produce the lore, or communal understanding, in a field, which in turn helps guide new research. Alternative procedures–confidence intervals, effect sizes, and meta-analysis–are discussed. Although these alternatives are not totally free from criticism either, they deserve more frequent use, without an unwise ban on NHSTs.