Being reliable: issues in determining the reliability and making sense of observations of adults with congenital deafblindness?

Prain et al. (2012) looked at how we check if two observers agree when they watch adults who are both deaf and blind. The team compared different math formulas used to score agreement. They wanted to see which formula gives a honest picture when the behavior is rare or subtle.

The paper is a methods guide, not an experiment. It shows how the same video clips can get very different 'reliability' scores depending on the formula picked.

Simple percent agreement made the data look great even when observers missed most events. Cohen's kappa told a truer, lower story. The authors warn that percent agreement can hide poor reliability when behaviors happen only once in a while.

They recommend always reporting Cohen's kappa and, when possible, having observers reach consensus before final coding.

Valluripalli Soorya et al. (2025) later showed the same danger in kids with severe ID from rare genetic syndromes. Standard autism tools failed because the kids' behaviors were too infrequent, backing up I et al.'s warning that rare events need tougher reliability checks.

Vos et al. (2013) echo the message from a different angle. Their methodology paper tells analysts to pick Yule's Q instead of simpler transitional probabilities when quantifying contingencies in sparse data. Both papers push the field to use more sensitive stats.

Mount et al. (2011) make a similar advance for motor data. They show random-coefficient modeling catches individual variability that regular GLM misses, just as Cohen's kappa catches observer drift that percent agreement hides.

Before you trust any observation log, demand both percent agreement and Cohen's kappa. If kappa is low even when agreement looks high, retrain observers or hold a consensus meeting. This single step keeps treatment decisions, functional analyses, and publication claims honest.