Evaluation of components of residential treatment by Medicaid ICF-MR surveys: a validity assessment.

Goldstein et al. (1991) checked if Medicaid ICF-MR surveys tell the truth. They compared survey scores for certified and non-certified units in large state homes for adults with intellectual disability.

The team looked at whether higher survey scores matched better client care. They also watched if staff acted differently when surveyors were in the building.

Surveys gave the same scores to both good and bad units. Certified units scored no better than non-certified ones.

Staff cleaned up only when surveyors arrived. This reactivity masked real quality gaps and made the survey useless.

Guercio et al. (2025) show a fix. They paid staff with small preferred items for turning in ≥80% of daily data sheets. Fidelity jumped in every house, proving you can get clean data if you motivate staff the right way.

Waller et al. (2010) saw the same reactivity risk. Month-end integrity checks clustered like schedule-induced scallops, warning that any announced observation can warp staff behavior.

Emerson et al. (2013) widen the lens. They caution that self-report and proxy data for people with ID often carry hidden validity flaws, backing the 1991 warning to doubt clipboard metrics.

Stop trusting clipboard surveys. If you need true quality data, use brief, unannounced behavioral samples or reward staff for accurate daily data like Guercio et al. did. Spread integrity checks across the month and keep them short to avoid the scallop trap. When you write reports, flag any measure that could be faked—your funding source and clients deserve real numbers, not theater.