What is it about?
Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate.
Featured Image
Photo by JOSHUA COLEMAN on Unsplash
Why is it important?
The multiple testing of hypotheses occurs in the most areas of science. For example, it occurs in clinical science, where researchers investigate whether a treatment affects multiple disease symptoms, and it occurs in psychology, where researchers investigate whether multiple groups of people hold different attitudes to one another. Multiple testing has been implicated in the replication crisis in science. In particular, it has been suggested that researchers who do not adequately correct their significance threshold, or alpha level, during multiple testing are at a greater risk of making Type I errors (incorrectly rejecting null hypotheses) and, consequently, publishing nonreplicable false positive results.
Perspectives
Rather than being based on the type of research situation (exploratory vs. confirmatory), my approach is based on the type of multiple testing. Specifically, I consider three types of multiple testing – disjunction testing, conjunction testing, and individual testing. I argue that an alpha correction for multiple testing is only necessary in the case of disjunction testing and not in the cases of either conjunction or individual testing. I explain when it is appropriate to undertake each type of multiple testing and, consequently, when it is appropriate to adjust alpha. Based on this explanation, I argue that researchers should not automatically assume that alpha adjustment is necessary during multiple testing.
Prof Mark Rubin
Durham University
Read the Original
This page is a summary of: When to adjust alpha during multiple testing: a consideration of disjunction, conjunction, and individual testing, Synthese, July 2021, Springer Science + Business Media,
DOI: 10.1007/s11229-021-03276-4.
You can read the full text:
Resources
Contributors
The following have contributed to this page
