Influences of response rate and distribution on the calculation of interobserver reliability scores.

Rolider et al. (2012) ran math checks on two common IOA formulas. They asked what happens when the same data have high response rates or most responses pile up at the end of each interval.

They fed fake data sets into exact-agreement IOA and total IOA. The goal was to see if the numbers still tell the truth when behavior gets fast or bunched.

Exact-agreement IOA cracked first. It dropped hard when responses came fast or landed late in the interval.

Total IOA stayed high and cheerful even when observers quietly disagreed. The paper warns that total IOA can hide real drift.

Cox et al. (2025) now gives you eight better tools. They say swap in precision, recall, or F1 instead of raw percent agreement. Their toolkit directly fixes the ceiling-effect trap that U et al. exposed.

Hausman et al. (2022) pushed the same lever further. They cut the number of IOA sessions and still saw the same rate-driven swings. This tells us the problem is stable across more or less data.

Jones et al. (1977) saw it coming. They already asked for Kappa and occurrence/non-occurrence splits long before U et al. proved raw percent agreement can lie.

Next time you track hand-flaps or vocal stereotypy that top 100 per minute, do not trust a glossy 95% total IOA. Flip to exact-agreement or Cox’s precision score. If the second number dips, retrain observers before you trust any trend line.