Using relative improvement over chance (RIOC) to examine agreement between tests: three case examples using studies of developmental coordination disorder (DCD) in children.
Swap kappa for RIOC when you check test agreement on rare disorders like DCD.
01Research in Context
What this study did
Cairney et al. (2011) wrote a how-to paper, not an experiment.
They show when to swap kappa for a different number called RIOC.
Three real DCD studies act as worked examples.
What they found
When very few kids have DCD, kappa says two tests hardly agree.
RIOC gives a clearer picture of how well the tests line up.
The authors give step-by-step math so you can repeat it.
How this fits with other research
Peng et al. (2026) meta-analysis pools 24 DCD exercise trials.
Their big effect sizes rely on good agreement labels—RIOC could sharpen those numbers.
Nelson et al. (1978) warn that autocorrelation warps visual versus statistical calls; John’s RIOC fix is another guard against hidden math traps.
Annable et al. (1979) found visual inspection agreement sits only at 0.61; using RIOC for extreme base rates could lift that reliability when you code DCD test results by hand.
Why it matters
If you screen for DCD, autism, or any low-base-rate condition, kappa can make two decent tests look awful. Run RIOC in Excel before you trash a tool. Sharper agreement numbers mean fewer kids mis-classified and better targets for your motor-skill programs.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Plug your last DCD screener data into the RIOC formula; see if your two best tools really disagree or if kappa lied.
02At a glance
03Original abstract
Although statistics such as kappa and phi are commonly used to assess agreement between tests, in situations where the base rate of a disorder in a population is low or high, these statistics tend to underestimate actual agreement. This can occur even if the tests are good and the classification of subjects is adequate. Relative improvement over chance or RIOC is a statistic that can correct for this bias; however, it is not yet commonly used in the health sciences or disability fields. In this paper, we introduce RIOC and demonstrate its application using the results from 3 previously published studies, all of which assessed the agreement between tests commonly used to identify children with a neurodevelopmental disorder known as developmental coordination disorder (or DCD). The results illustrate the differences between kappa and RIOC under conditions where the distribution of scores in a 2 × 2 table is unbalanced. The introduction of this statistic in the area of developmental disabilities research is encouraged.
Research in developmental disabilities, 2011 · doi:10.1016/j.ridd.2010.09.001