Further evaluating interobserver reliability and accuracy with and without structured visual‐inspection criteria
Give every new BCBA the Roane visual-inspection checklist—accuracy shoots up even if agreement only inches forward.
01Research in Context
What this study did
Cox et al. (2021) asked new BCBAs to look at functional-analysis graphs.
Half got the Roane visual-inspection checklist. Half got no form.
The team then scored how close each novice came to expert answers.
What they found
Accuracy jumped when the checklist was in hand.
Agreement between novices only nudged upward.
In short, the form fixed more wrong answers than it created matches.
How this fits with other research
Guerrero et al. (2022) repeated the idea with mealtime graphs and saw the same pattern: accuracy up, but only small reliability gains.
Morantz et al. (2022) went further and coded the checklist into an R script. The robot agreed with experts 81% of the time, showing the rules still work when a human is not holding the pencil.
Machado et al. (2021) trained observers to score behavior videos at 5× speed. Both studies show that brief, clear rules can slash error without long lectures.
Why it matters
You can hand the Roane sheet to a new hire today. One page lifts their FA-call accuracy right away. Keep using it yourself, or let the free R script do the math. Either way, you spend less time fixing graphs and more time treating clients.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Print the Roane checklist, slide it into each trainee’s binder, and have them use it on the next five FA graphs you review together.
02At a glance
03Original abstract
AbstractVisual inspection is the primary method of interpreting functional analysis (FA) outcomes, even though it has occasionally been criticized for producing low levels of interobserver agreement. Researchers have addressed this issue by creating structured visual‐inspection criteria to guide visual inspection of FA outcomes (e.g., Hagopian et al., 1997, https://doi.org/10.1901/jaba.1997.30‐313; Roane et al., 2013, https://doi.org/10.1002/jaba.13). The purpose of the current study was to systematically replicate and extend Study 1 of Roane et al. (2013, https://doi.org/10.1002/jaba.13). We did this by evaluating the reliability and accuracy of 15 novice participants’ visual inspection of 84 FA graphs with and without the modified visual‐inspection criteria developed by Roane et al. Accuracy was markedly higher when participants used the modified visual‐inspection criteria relative to when they used traditional visual‐inspection strategies, while we observed more modest increases in reliability coefficients. Results are discussed in the context of practical and clinical implications of the modified visual‐inspection criteria and suggestions for future research.
Behavioral Interventions, 2021 · doi:10.1002/bin.1793