Interobserver agreement: A preliminary investigation into how much is enough?

Hausman et al. (2022) re-checked old functional-analysis tapes. They clipped the tapes at different places. Then they asked: does IOA change if we score 5, 10, or 20 sessions?

The team used trained observers and clean clinic rooms. They wanted to see if a magic session number gives better agreement.

IOA stayed almost the same no matter how many sessions they kept. The only thing that moved the score was how often the behavior happened.

In short, extra sessions did not clean up the numbers. Response rate, not session count, drove the differences.

Rolider et al. (2012) already warned that exact-agreement IOA jumps around when behavior is fast. Hausman’s data now show the jump comes from rate, not from grabbing more sessions.

Cox et al. (2025) just gave us new math—precision, recall, F1—to replace old percent agreement. Pair that with Hausman’s finding and you can run fewer sessions and still report sharper metrics.

Essig et al. (2023) counted papers and saw half skip fidelity IOA. Hausman answers the quiet worry behind that gap: you don’t need a pile of extra IOA sessions to feel safe.

Stop collecting “just-in-case” IOA sessions. Train your observer well, control the room, and score until the rate looks stable. You save time and still meet the journal standard.