An exploration of the interrater agreement of visual analysis with and without context

Ford et al. (2020) asked 60 behavior analysts to look at single-case graphs.

Half the group saw only the graph. The other half got the same graph plus a short note about the client, setting, and target behavior.

Everyone decided whether a functional relation was present. The authors then checked how often the two groups agreed with each other.

Extra context did not change agreement. Both groups reached the same yes-or-no decision on six of seven graphs.

Inter-rater agreement stayed high no matter which packet the raters received.

In short, the graph alone was enough for experts to reach a reliable decision.

Lancioni et al. (2008) once warned that visual inspection of FA graphs is shaky. Ford’s team shows the act itself can be reliable when the graph is clear; the earlier low numbers likely came from hard-to-read data, not from missing context.

Taylor et al. (2022) later found that computer rules matched visual inspection on about four out of five AB graphs. Together the two studies give you a simple recipe: trust your eyes for clear graphs, and keep an objective algorithm handy for the messy ones.

Stolz (1977) showed that old JABA papers rarely checked reliability more than once per condition. Ford’s work keeps that issue alive—agreement is only high when the display is clean, so keep collecting reliability every phase.

You can stop hunting for extra client details before you judge a graph. Focus on making the display clear instead: phase lines, equal axes, and steady trend. If the data path is messy, pair your visual call with a conservative dual-criteria or machine-learning check. This saves time and still protects decision quality.