Brief Report: An Exploratory Study of the Diagnostic Reliability for Autism Spectrum Disorder.
Clinicians looking at the same ADOS video agree on autism only one-third of the time, so always verify outside diagnoses with your own observation.
01Research in Context
What this study did
Twenty-seven clinicians watched ADOS videos and gave autism diagnoses. The researchers counted how often the doctors agreed.
They wanted to see if different clinicians reach the same decision when they all view the same recorded ADOS session.
What they found
Only one in three videos earned the same diagnosis from every clinician. Perfect agreement happened just 33 percent of the time.
When the team used a stricter kappa rule, only 24 percent of cases reached good or excellent agreement.
How this fits with other research
Lancioni et al. (2009) saw the same problem in the field. Community teams agreed on autism only 45 percent of the time, showing poor reliability is not new.
Stadnick et al. (2015) looks like a contradiction. Their community clinics found ADOS lined up with final diagnosis in most cases. The difference: they compared ADOS to a full work-up, not to other clinicians watching the same tape.
Sutherland et al. (2025) moved the ball forward. Using a tight telehealth protocol, two doctors agreed on 19 of 21 toddlers — 90 percent. Good structure and training can fix the reliability gap the 2017 paper exposes.
Why it matters
Low agreement means the same child could leave with two different labels. For BCBAs, this warns us to treat outside diagnoses as tentative until you run your own ADOS or observe the client directly. Push for second opinions when reports conflict, and document your own data before starting treatment.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Schedule a brief direct observation or ADOS module before writing the treatment plan whenever a referral diagnosis feels unclear.
02At a glance
03Original abstract
Previous research shows inconsistency in clinician-assigned diagnoses of Autism Spectrum Disorder (ASD). We conducted an exploratory study that examined the concordance of diagnoses between a multidisciplinary assessment team and a range of independent clinicians throughout Australia. Nine video-taped Autism Diagnostic Observation Schedule (ADOS) assessments were collected from two Australian sites. Twenty-seven Australian health professionals each observed two video-recordings and rated the degree to which the individual met the DSM-5 criteria for ASD. There was 100% agreement on the diagnostic classification for only 3 of the 9 video clips (33%), with the remaining 6 clips (66%) reaching poor reliability. In addition, only 24% of the participating clinicians achieved 'good' or 'excellent' levels of agreement (Cohen's kappa > 0.6) with the original ASD assessment. These findings have implications for clinical guidelines for ASD assessments.
Journal of autism and developmental disorders, 2017 · doi:10.1007/s10803-017-3054-z