The influence of data characteristics on interrater agreement among visual analysts

Wolfe et al. (2023) asked 36 BCBAs to look at 120 single-case graphs on a computer. Each graph showed six different data features: trend, effect size, variability, overlap, data points per phase, and phase length.

The analysts rated whether they saw a clear treatment effect. The team then checked which features made raters agree or disagree most strongly.

Steep trend and large effect size hurt agreement the most. High variability also lowered agreement, but less than trend and effect size. The other three features—overlap, points per phase, and phase length—hardly mattered.

In plain words, when a graph shoots up or down sharply, or when dots bounce widely, BCBAs are more likely to argue about what they see.

McGonigle et al. (1982) first showed that experts ignore variability when they eyeball graphs. Wolfe et al. (2023) now prove that variability still splits raters, even if they think they ignore it. The older and newer studies line up: variability quietly sows doubt.

Diller et al. (2016) also ran a survey and found low agreement among BCBAs judging multielement designs. Both papers blame variability and trend, giving the same warning across very different graph types.

Peltier et al. (2024) tried shrinking the y-axis or removing dot boxes and saw no boost in confidence. Wolfe et al. (2023) add that you cannot fix disagreement with layout tricks if the data itself are steep or jumpy.

Before you show a graph to a team, check the slope and bounce. If the line rockets up or the dots scatter wide, add more stable baseline days or tighten the intervention first. Cleaner data make team decisions faster and reduce wasted time arguing over pictures.