Beyond matching on the mean in developmental disabilities research.
Stop trusting “groups were matched” until you see the spread, shape, and item-level data.
01Research in Context
What this study did
Facon et al. (2011) wrote a how-to paper, not an experiment.
They looked at how researchers build “matched” groups in developmental-disability studies.
The team warned that only checking if the group averages are the same hides big problems.
What they found
The paper says stop at the mean and you can miss wide scatter, skewed scores, and single kids who throw the whole study off.
They tell readers to also eyeball the spread, the shape of the curve, and item-by-item profiles before calling two groups “equal.”
How this fits with other research
Ganz et al. (2004) set the stage seven years earlier. They already said “don’t trust age-equivalent scores” and “use tougher alpha levels.” Bruno brings the same worry but adds new graphs to inspect.
Flapper et al. (2013) pick up the baton two years later. They turn Bruno’s “look at the spread” into numbers: report standardized mean differences and variance ratios. The three papers form one growing recipe.
Jarrold et al. (2004) run a parallel critique in autism work. They swap IQ-only matching for control-task matching, showing the issue crosses diagnoses.
Why it matters
Next time you read—or design—a study with matched groups, flip past the t-test table. Plot the raw scores, check the range, and demand variance numbers. If you review a grant or journal article, ask for effect-size metrics like Flapper et al. (2013) suggest. Better matching means truer cause-effect claims for the kids we serve.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Open the last group study on your desktop and graph the control and treatment scores side-by-side to eyeball variance.
02At a glance
03Original abstract
The matching of groups is a traditional way to control for confounding variables in developmental disabilities research. The equivalency of means across groups is routinely checked for these variables, but not the homogeneity of their variances or the shapes of their distributions. In the present paper, it is argued that group matching can go seriously wrong unless it directly confronts the distributional concerns by the use of well-known statistical indices and very simple graphical displays of the distributions. The question of the equivalency of item response profiles is also addressed since two participants or two groups of participants can obtain the same overall score on the matching variable by passing different items. In this case, the matching cannot be considered satisfactory because of poor concordance between the molar (overall score) and molecular (item scores) levels of matching. Angoff's Delta plot method, a statistical approach for detecting differential item functioning across small groups is described. It is promising as a simple way to prove whole test/individual item correspondence and, in addition, a useful tool for making post hoc statistical analyses at the item level on the dependent variables.
Research in developmental disabilities, 2011 · doi:10.1016/j.ridd.2011.07.029