Observer reliability as a function of circumstances of assessment.

The team watched how observers act when they know someone is checking their data.

They changed three things: telling or not telling observers about checks, having a boss in the room or not, and letting pairs grade themselves or using outside pairs.

Then they looked at how high the IOA numbers looked under each setup.

IOA jumped when observers knew they were being checked that day.

Scores also rose when no boss watched the session and when partners checked each other instead of strangers.

In short, the same video could give very different reliability scores depending on the check setup.

Normand et al. (2023) warn that researchers who wear both clinician and scientist hats can bias data; Branch et al. (1977) show even well-meaning observers can tilt numbers if the check system is loose.

Critchfield et al. (2003) found that a “reward” can secretly punish; here, a “check” can secretly inflate. Both papers scream the same message: tiny procedural details swing results.

Schmidt et al. (1969) got clean data by using clear, reset-based rules in class; N et al. prove you need equally clear, blind, cross-pair rules to get honest IOA.

You can’t trust high IOA if your observers know today is check day, sit alone, and swap sheets with a friend. Build blind spot checks, rotate outside pairs, and drop in unannounced. Honest data starts with honest measurement.