Interrater agreement between visual analysts of single-case data: a meta-analysis.

The team pooled 23 older studies where pairs of BCBAs judged the same single-case graphs.

They asked: how often do two people agree that a line is flat, rising, or falling?

The meta-analysis covered 1,000+ graphs from classrooms, clinics, and labs.

Across all studies, agreement averaged only a large share.

That means one in four graphs got different yes/no calls from two trained viewers.

Agreement dropped even lower when trends were small or data were messy.

Lam et al. (2011) saw the same risk with numbers: switching from continuous to 10-s partial interval recording inflated IOA scores. Both papers warn that the method you pick can hide poor agreement.

Evers et al. (2020) found the same a large share overlap between two popular ASD parent interviews. Together the studies show moderate agreement is common across very different tools—graphs, checklists, or interviews.

Samyn et al. (2015) showed questionnaires and lab tasks measure different skills, so you can’t swap them. Likewise, Jennifer et al. remind us that eye-balling graphs without rules is not the same as using a defined rubric.

Before you write “intervention worked” in a report, pair up and score the graph with a coworker. Use a simple rule sheet: trend must cross three data points, no overlap with baseline, etc. Add median lines or split-middle lines to the chart so both raters look at the same cue. These quick steps push your team past the shaky a large share mark and give parents, teachers, and payers clearer proof.