Measuring the reliability of observational data: a reactive process.

Wildemann et al. (1973) asked a simple question: does telling observers you will check their agreement change how they code? They ran two conditions. In one, observers knew a second coder would compare notes. In the other, the check stayed hidden.

The team then looked at how often the two coders matched. They wanted to see if the simple act of announcing a reliability check skewed the numbers.

When observers knew they were being checked, their agreement scores shot up. The covert condition gave lower numbers. The authors argue the lower figures are the honest ones.

In short, the warning cue alone pushed coders toward the known assessor's data.

Harris et al. (1978) extends the warning. They showed that even when reliability checks stay in place, watching a teacher hand out praise can nudge observers to over-count eye contact. Three of six coders inflated scores after seeing rewards delivered.

Palmer et al. (2018) echoes the same theme with college students. Just having an experimenter in the room cut off-task behavior in half. Both papers shout the same message: measurement itself changes what you measure.

Fahmie et al. (2013) built a new autism scale and reported decent inter-rater reliability. Yet that claim is open to the same reactivity G et al. exposed; if the coders knew a check was coming, their agreement may be artifically rosy.

Next time you train a new RBT to collect data, run covert reliability trials at unannounced times. Rotate second observers quietly into sessions. Report both the open and hidden agreement numbers in your treatment reports. This small step keeps your data honest and your clinical decisions solid.