Randomization Tests for Single Case Designs with Rapidly Alternating Conditions: An Analysis of p-Values from Published Experiments
Set alpha at .05 when you run a randomization test on rapidly alternating single-case data and your p-value will usually match expert visual judgment.
01Research in Context
What this study did
Weaver et al. (2019) looked at p-values from randomization tests on published alternating-condition single-case graphs. They sorted graphs that experts called 'clear effects' from graphs experts called 'no effect.'
They wanted to see if the p-values line up with visual calls when alpha is set at .05.
What they found
Strong-effect graphs gave mostly low p-values; no-effect graphs gave mostly high p-values. A cutoff of .05 matched visual analysis most of the time.
Randomization tests can give you a yes-no answer that agrees with what your eyes see.
How this fits with other research
Ferron et al. (2023) extend the same logic to changing-criterion designs. You can now randomize criterion shifts and still get a clean p-value.
Manolov (2019) ran simulations on alternating-treatment data and also endorsed randomization tests, but added ALIV to quantify the size of the jump.
Smit et al. (2019) took the opposite road: they built a visual checklist instead of a p-value. Both papers target the same ATD graphs, one with stats, one with visuals.
Why it matters
If you run multielement or ABAB designs, you can now back up visual calls with a randomization p-value at .05. Free tools from Levin et al. (2019) make the randomization step take five minutes. Add the step to your next study and give reviewers both eyes and numbers.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Plug your last multielement graph into a free randomization-test Excel sheet and compare the p-value to your visual call.
02At a glance
03Original abstract
Two common barriers to applying statistical tests to single-case experiments are that single-case data often violate the assumptions of parametric tests and that random assignment is inconsistent with the logic of single-case design. However, in the case of randomization tests applied to single-case experiments with rapidly alternating conditions, neither the statistical assumptions nor the logic of the designs are violated. To examine the utility of randomization tests for single-case data, we collected a sample of published articles including alternating treatments or multielement designs with random or semi-random condition sequences. We extracted data from graphs and used randomization tests to estimate the probability of obtaining results at least as extreme as the results in the experiment by chance alone (i.e., p-value). We compared the distribution of p-values from experimental comparisons that did and did not indicate a functional relation based on visual analysis and evaluated agreement between visual and statistical analysis at several levels of α. Results showed different means, shapes, and spreads for the p-value distributions and substantial agreement between visual and statistical analysis when α = .05, with lower agreement when α was adjusted to preserve family-wise error at .05. Questions remain, however, on the appropriate application and interpretation of randomization tests for single-case designs.
Perspectives on Behavior Science, 2019 · doi:10.1007/s40614-018-0165-6