abstract Eric-Jan Wagenmakers

Eric-Jan Wagenmakers (Department of Psychological Methods, University of Amsterdam, The Netherlands)

The diagnosticity of a p value

In many fields of study, the evidence against the null hypothesis is summarized by the p value. By decree, a p value lower then .05 allows researchers to reject the null hypothesis. Here I describe a simple analysis that quantifies the maximum diagnosticity of the p value. The analysis considers how likely an observed p-value is to occur under H0 versus under H1. An upper bound on this likelihood ratio can be obtained by cherry-picking the best possible H1 (Sellke et al., 2001). The results are dramatic, and highlight that p values higher than .01 can never be very diagnostic. This demonstrates that p values are "violently biased against the null hypothesis" and suggests that the common “p < .05” rule for significance is too lenient.