Interobserver agreement and procedural fidelity: An odd asymmetry

Essig et al. (2023) read every experiment in JABA and Behavior Analysis in Practice from 2017 through 2021. They counted how many papers reported procedural fidelity and how many checked if two observers agreed on fidelity scores.

They also noted how often studies reported interobserver agreement (IOA) on client behavior. The goal was to see if researchers treat fidelity data the same way they treat behavior data.

Only half of the experiments mentioned procedural fidelity at all. When fidelity was tracked, fewer than 18% of papers showed IOA on those fidelity numbers.

In contrast, almost every paper showed IOA on client behavior. The gap shows we guard against observer error for behavior but not for how well we ran the procedure.

Bergmann et al. (2023) looked at the same journals and years and also found weak fidelity reporting. Their paper adds a handy checklist, while Essig et al. highlight the missing IOA layer. Together they form a full picture of the problem.

Jones et al. (1977) warned decades ago that IOA reports were too thin. Essig et al. show the field still skips IOA on fidelity, proving the old warning still bites.

Cox et al. (2025) give new IOA tools like precision and recall. These metrics could close the gap Essig et al. found, making fidelity checks as solid as behavior checks.

If we do not check fidelity IOA, we cannot be sure the treatment was done the same way every time. That weakens both internal and external validity. Next time you write or review a study, add a second observer to at least 20% of fidelity sessions and report the agreement. The extra step takes minutes and saves your data from silent drift.