A quantitative analysis of accuracy, reliability and bias in judgements of functional analyses
Doctoral BCBAs still misread FA graphs, so pair visual review with a decision rule or second rater.
01Research in Context
What this study did
Rader et al. (2021) sent fake functional-analysis graphs to BCBA-Ds.
They asked the experts to judge if problem behavior was higher in one condition.
Then they checked how often the experts agreed with the true data pattern.
What they found
Even doctoral-level BCBAs were wrong or split on many graphs.
Reliability between experts was shaky, just like with newer BCBAs in older work.
How this fits with other research
Mount et al. (2011) already showed that high variability makes any viewer less sure.
Falligant et al. (2020) later counted false alarms with dual-criteria rules and found the same weak spots.
The new twist: letters after your name do not fix the problem.
Why it matters
If experts misread FA graphs, treatment choices can be wrong.
Add a decision aid like DC/CDC rules or a short team review before you pick the function.
Your eyes alone are not enough.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Open your last FA plot, apply a dual-criteria rule, and compare the result to your original call.
02At a glance
03Original abstract
Functional analysis can be considered a diagnostic assessment that behavior analysts use to determine behavioral function. Such a diagnosis ultimately requires a yes or no decision (i.e., a variable maintains a behavior, or it does not) that is determined by both subjective (clinical judgement) and objective (data) variables. Accurate and reliable identification of function is essential for successful treatment, yet behavior analysts' interpretation of data relies on their ability to detect visual differences in graphed data. Some research indicates that behavior analysts have questionable reliability in their visual analysis. To further examine the reliability, accuracy, and bias in visual analysis of functional analysis graphs, we simulated functional analysis results and surveyed 121 BCBA-Ds experienced in visual analysis. We then examined reliability of responses and used a signal detection theory approach to analyze accuracy and bias. Findings suggest that reliability and accuracy of judgements are questionable, and exploration of decision aids is warranted.
Journal of the Experimental Analysis of Behavior, 2021 · doi:10.1002/jeab.711