Statistical Inference as Severe Testing, by Deborah Mayo. Statistics wars, P-values, and more.
What is it about?
Over-riding themes that this reviewer identifies are: 1) Warnings against practices that lead to BENT science (Bad Evidence, No Test). These include cherry picking, multiple testing, artificiality of experiments, publication bias, and so forth. 2) An elaboration of Karl Popper's ideas of severe testing, as they relate to scientific hypotheses, here within a null hypothesis significance testing (NHST) context. 3) A distinction between the philosophies 'Probabilism' (primarily Bayesian); `Performance' (where the sampling distribution has a central role); and her own philosophy that she terms `Probativeness' (identified as performance within a severe testing context). 4) A defense of the role of P-values that leaves little or no room for investigation of the implications the choice of prior. In one class of examples that are discussed, the author is unwilling to allow a data-based choice. 5) Extensive discussion of the history, extending through to recently published work, from the author's own null hypothesis significance testing perspective.
Why is it important?
How, based on available evidence, are defensible scientific judgments properly made? There can be no doubting the importance to scientific work of the issues addressed in this book. This review addresses a number of points that have not attracted the attention that they deserve in other reviews that I have seen. I argue that replication studies, to date mainly those mounted by the University of Virginia Center for Open Science (COS), have a much greater importance than Mayo allows them. They offer insights that are otherwise unavailable into the way that research and publication processes function. Those insights are of central importance in any discussion of scientific processes.
The following have contributed to this page: John Maindonald
In partnership with: