Observer agreement, credibility, and judgment: some considerations in presenting observer agreement data.

Jones et al. (1977) wrote a think-piece, not an experiment. They looked at how ABA papers were showing observer agreement. Most authors only gave one number: total percent agreement. The team said that single number hides important facts.

They urged writers to add two more checks. Report separate scores for times the behavior happened and did not happen. Also add Kappa or Phi to correct for chance agreement.

The paper found no new data. Instead it found a reporting gap. Readers could not tell if high agreement came from real consensus or from lucky guesses. The authors warned that weak reports hurt the field’s credibility.

Harris et al. (1978) answered the call the very next year. They gave a weighted formula that mixes occurrence and non-occurrence scores. Cox et al. (2025) went even further. Their 2025 paper replaces percent agreement with eight new indices like precision and recall for cases with very low or very high behavior rates.

Hausman et al. (2022) tested a related worry: do we need lots of extra IOA sessions? Their re-analysis showed stable coefficients come quickly with well-trained observers, so you can follow R et al.’s advice without piling on sessions.

Essig et al. (2023) widened the lens. They audited recent JABA and BAP articles and found that half still skip procedural-fidelity IOA. This gap keeps the credibility problem alive almost fifty years after the target paper sounded the alarm.

Next time you write a report, do not stop at 95% agreement. Add occurrence agreement, non-occurrence agreement, and Kappa or Phi. These extra numbers take five minutes to calculate and show reviewers you did not hide disagreement. The habit boosts your study’s trust and keeps pace with newer indices if you later need them.