The behavior equivalence problem in within-subject treatment comparisons.

Stoddard et al. (1988) wrote a how-to paper, not an experiment. They asked: how can we compare two teaching methods on the same worker if the jobs are different?

Their fix is to build small groups of tasks that feel equally hard. They call these groups item cohorts. You pick one task from each cohort for each method you want to test.

The paper gives a step-by-step recipe. First, rank every task by how many correct responses it takes to finish. Next, bundle tasks with close scores into cohorts. Last, draw one task per cohort for each treatment arm.

This keeps difficulty the same across treatments, so any change you see is more likely from the teaching method, not from the job being easier or harder.

McMillan (1973) warned that sequence effects can fool you in within-subject designs. T et al. answer that warning with a practical tool: item cohorts.

McLennan et al. (2008) later showed kids shift to easier work even when rewards stay equal. Their data back up the warning T et al. tried to solve.

Cariveau et al. (2021) scanned 30 years of adapted alternating treatments studies. They found most papers skip any difficulty check. They echo T et al.'s call and extend it to academic targets like sight words.

Next time you run an alternating treatments design, take 15 minutes to build item cohorts. List your targets, score them for difficulty, then match them across conditions. This tiny step blocks a major threat to internal validity and makes your data cleaner without extra sessions.