A probability-based formula for calculating interobserver agreement.
Replace percent agreement with the 1977 chance-corrected formula for more honest IOA scores.
01Research in Context
What this study did
The authors built a new math formula for interobserver agreement.
It counts both times observers agreed a behavior happened and times they agreed it did not.
The formula also removes the agreements that could happen just by chance.
What they found
The new formula gives a cleaner number than simple percent agreement.
It stops you from thinking your data are perfect when observers might just be lucky.
How this fits with other research
Jacobs (2019) later pushed for randomization tests instead of old t-tests.
Both papers want stronger numbers in single-case work.
Manolov et al. (2022) gave a free visual tool to judge if effects repeat.
That tool pairs well with the 1977 formula: first check observer agreement, then check replication.
Lanovaz et al. (2017) set rules for phase length to keep false alarms low.
Using the 1977 formula plus Lanovaz rules gives you both clean data and clean decisions.
Why it matters
Stop using plain percent agreement today. Plug the 1977 formula into Excel or any stats sheet. Your IOA will now tell you if observers truly agree or are just guessing alike. Cleaner IOA means cleaner functional analyses and cleaner treatment choices.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Open your last five session sheets, recalculate IOA with the 1977 formula, and flag any file that drops below a large share.
02At a glance
03Original abstract
Estimates of observer agreement are necessary to assess the acceptability of interval data. A common method for assessing observer agreement, per cent agreement, includes several major weaknesses and varies as a function of the frequency of behavior recorded and the inclusion or exclusion of agreements on nonoccurrences. Also, agreements that might be expected to occur by chance are not taken into account. An alternative method for assessing observer agreement that determines the exact probability that the obtained number of agreements or better would have occurred by chance is presented and explained. Agreements on both occurrences and nonoccurrences of behavior are considered in the calculation of this probability.
Journal of applied behavior analysis, 1977 · doi:10.1901/jaba.1977.10-127