Interobserver Agreement among a Staff Member and Visitors at a Wolf Sanctuary
Untrained helpers can hit 80% IOA on one easy behavior with a free app, but adding a second behavior drops their reliability.
01Research in Context
What this study did
Researchers asked everyday zoo visitors to watch wolves and tap a free phone app when they saw a target behavior.
Half the guests tracked one behavior, such as pacing. The other half tracked two behaviors at once, like pacing and howling.
A trained staff member watched the same wolves and scored the behaviors for comparison.
What they found
Visitors who watched one behavior agreed with the expert at least 80% of the time.
When guests had to watch two behaviors, their agreement stayed high, but a stricter kappa score dipped below the good range for some.
The app worked best when the rules were short and the behavior was easy to see.
How this fits with other research
Marano et al. (2020) also showed that people with little training can judge behavior well, as long as they first watch and rate short clips.
Yarzebski et al. (2024) proved that a quick video model can teach caregivers near-perfect technique, matching the low-cost spirit of the wolf study.
Rice et al. (2009) quadrupled staff greeting accuracy with a five-minute talk and praise, echoing how simple instructions gave visitors 80% accuracy here.
García‐Leal et al. (2019) warns that extra effort can warp data; likewise, asking guests to score two behaviors at once lowered their kappa scores.
Why it matters
You can hand a tablet to a parent, volunteer, or new tech and collect solid data on a single, clear behavior. Keep the definitions short, add photos, and check the first few sessions. If you need two behaviors, train a bit more or split the job between two people.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Pick one clear target behavior, load a free tally app with a picture prompt, and have a parent or aide collect five minutes of data while you watch; compare counts and adjust definitions until you both hit 80%.
02At a glance
03Original abstract
The purpose of this study was to determine whether untrained visitors at a wolf sanctuary could collect behavioral data, within an acceptable range of interobserver agreement (IOA), using a freely available instrument on their smartphones. Two independent groups of untrained visitors continuously recorded one target behavior only or two target behaviors simultaneously. Observations occurred during a 50-min educational presentation with multiple wolves. Whereas IOA was at or above 80% for all participants, kappa was below the acceptable range of 0.6 or greater for some participants observing two target behaviors simultaneously. • Clinicians may consider the use of untrained observers for limited data collection purposes • Simple recording rules or pictures of conspicuous target behaviors may facilitate acceptable interobserver agreement • Clinicians may consider the use of a similar smartphone data collection instrument for untrained observers • Clinicians should be cautious of any competing smartphone behaviors that may interfere with the behavior of observing a target
Behavior Analysis in Practice, 2024 · doi:10.1007/s40617-024-00959-5