Visual analysis of data in a multielement design

Diller et al. (2016) asked 36 BCBAs to look at 60 multielement graphs. Each graph showed two conditions side-by-side. The raters judged whether clear experimental control was present.

The team then mixed in different levels of variability, trend, and mean shifts. They wanted to see which features made raters agree or disagree.

Agreement among the BCBAs was low. Only about two-thirds of the pairs picked the same yes-or-no verdict on a given graph.

High variability within a condition and steep trends across time made raters less sure. A big mean shift between conditions pulled them toward saying 'yes, control is clear.'

Wolfe et al. (2023) later tested 1,488 ABAB graphs and saw the same culprits: trend and variability still wrecked agreement. Their larger set shows the worry is not just a multielement quirk.

Dowdy et al. (2024) moved to functional-analysis graphs and found that stretching the x-to-y axis ratio could also sway visual calls. Together the three studies warn that both data features and how we draw the graph can bias our eyes.

Wolfe et al. (2018) offers a partial fix. Their conservative dual-criterion (CDC) method agreed well with expert eyes on many ABAB graphs. When agreement is shaky, running CDC alongside visual inspection can give you a second, numbers-based opinion.

Your visual check is still the gold standard in the field, but this paper shows it can be a scratchy ruler. Before you stake a treatment decision on a multielement graph, run a quick CDC or count-based aid if variability or trend is high. Share the graph with a colleague and note where you differ. Simple steps like these can turn subjective eyeballing into a team sport and save you from a false positive call.