Analyzing single-case studies: sound statistical conclusions begin with the design

What is it about?

Single-case studies follow one person over time, measuring a behavior or symptom repeatedly before and after an intervention begins. They are widely used in applied behavior analysis, special education, and clinical psychology. Two features make these data hard to analyze: the series are short, and consecutive measurements tend to resemble one another, a property called serial dependence. Many common analyses ignore it. This study used computer simulation to test six widely used methods (the nonoverlap indices NAP and Tau-U, Hedges’s g, piecewise regression, a randomization test, and an approximate Bayesian regression model) across 218 conditions that varied series length, the strength of serial dependence, baseline trend, and the type of treatment effect. Each method was judged on three questions. Does it avoid false alarms when the treatment has no effect? Do its 95% intervals contain the true value as often as they claim? And how often does it detect a real effect when all methods are held to the same false-alarm rate?

Featured Image

Photo by UX Indonesia on Unsplash

Photo by UX Indonesia on Unsplash

Why is it important?

The choice of method can change the conclusion of a single-case study. Under strong serial dependence, the two regression-based methods (piecewise regression and the Bayesian model) reported a treatment effect in as many as 27% to 37% of series in which none existed, instead of the intended 5%, and their 95% intervals captured the true value as rarely as 62% of the time: an illusion of precision. Their apparent advantage in detecting effects came largely from those false alarms. Compared at the same actual false-alarm rate, their advantage for immediate changes in level largely disappeared; gradual changes in trend remained the hardest to detect, and there the randomization test, with the statistic studied, stayed less sensitive. The randomization test looked weak mainly because it was applied to studies in which nothing had been randomized. When the intervention start point was chosen at random, its false-alarm rate matched the attainable level (the largest rate at or below 5% that the design permits, 4.2% with 30 observations) at every degree of dependence. The practical message is to randomize the start point where possible, plan for 30 or more observations, use randomization tests for confirmatory decisions, describe the consistency of change with nonoverlap indices, and use autocorrelation-aware regression for detailed estimation.

Perspectives