A comparison of reliability measures for continuous and discontinuous recording methods: inflated agreement scores with partial interval recording and momentary time sampling for duration events.
Ten-second partial interval or momentary time sampling pumps up your IOA—use continuous duration recording when you care about how long behavior lasts.
01Research in Context
What this study did
The team compared three ways to score the same duration events. They used continuous duration recording, 10-s partial interval recording (PIR), and 10-s momentary time sampling (MTS).
They asked: do these methods give the same inter-rater agreement (IOA) number? They ran the numbers on pretend data sets that looked like real behavior streams.
What they found
PIR and MTS always gave higher IOA scores than the continuous record. The shorter the behavior lasted, the bigger the fake boost.
In some cases the inflation was huge. A true a large share agreement could look like a large share just by switching to PIR or MTS.
How this fits with other research
Stephenson et al. (2015) pooled 19 visual-analysis studies and also found so-so agreement (a large share). Their work shows reliability problems are everywhere, not just with duration coding.
Samyn et al. (2015) proved questionnaires and performance tasks measure different things. Lam et al. (2011) now show that even within direct observation, the recording rule you pick can fake a agreement jump.
Together the papers scream the same message: the tool and the rule matter as much as the behavior you are trying to see.
Why it matters
If you use 10-s PIR or MTS to track how long a child stays on task, your IOA column will look prettier than it should. That can hide observer drift and waste treatment time. Stick with continuous duration recording when the goal is minutes engaged. If you must use PIR or MTS for logistics, cut the IOA bar to a large share or lower and double-check with occasional continuous samples.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Open last week’s data sheets—if you used 10-s PIR for duration, re-score one session with continuous recording and compare the real IOA.
02At a glance
03Original abstract
The authors evaluated the extent to which interobserver agreement (IOA) scores, using the block-by-block method for events scored with continuous duration recording (CDR), were higher when the data from the same sessions were converted to discontinuous methods. Sessions with IOA scores of 89% or less with CDR were rescored using 10-s partial interval recording (PIR) and 10-s momentary time sampling (MTS). Results indicated that IOA scores for 10-s PIR and 10-s MTS were consistently higher than IOA scores based on CDR for the same sessions. Specifically, 10-s MTS provided higher overestimations for low-duration events, whereas 10-s PIR produced higher overestimations for moderate- and high-duration events. Implications for researchers and clinicians are briefly discussed.
Behavior modification, 2011 · doi:10.1177/0145445511405512