Thematic and textual analysis methods for developing social validity questionnaires in applied behavior analysis
Count words instead of sorting themes when you build social-validity questionnaires—Anderson shows it is more reliable and just as valid.
01Research in Context
What this study did
Anderson et al. (2022) compared two ways to build Likert-type social-validity questionnaires.
One way was thematic analysis, where coders group similar ideas into themes.
The other way was textual analysis, where coders count exact words and phrases.
They checked which method gave more reliable ratings and still covered the same content.
What they found
Textual analysis gave higher inter-rater reliability.
Both methods produced questionnaires with equal content validity.
In plain words: counting words led to more agreement without losing meaning.
How this fits with other research
Lichtlé et al. (2022) used traditional thematic analysis to pick quality-of-life items for preschoolers with autism.
Anderson’s textual method offers a more reliable alternative for the same task.
Szempruch et al. (1993) warned that social-validity ratings are often vague.
Anderson answers that call by giving a reproducible, number-based coding plan.
Costello et al. (2022) showed visual inspection alone is unreliable and urged quantitative supplements.
Anderson mirrors this move by replacing subjective theme picking with word counts.
Why it matters
When you next craft a social-validity scale, skip the long theme debates.
Use textual analysis: list key words, count them, and build items straight from the numbers.
You will finish faster and your team will agree more often, giving stronger evidence that stakeholders truly value your intervention.
Want CEUs on This Topic?
The ABA Clubhouse has 60+ free CEUs — live every Wednesday. Ethics, supervision & clinical topics.
Join Free →Open your last survey draft, highlight the key stakeholder quotes, count the top five repeated words, and turn those into your Likert items.
02At a glance
03Original abstract
AbstractSocial validity is often defined as the degree to which an intervention has value to the community that it effects, but it is seldom reported in the literature. Most social validity questionnaires are purposely created by the authors of the study and often lack a description of scale development process. The purpose of this study was to evaluate methods for the development of a Likert‐type social validity scale. Caregivers of children who took part in a study on behavioral treatments for pediatric feeding disorders were part of an initial interview to inform scale development. We analyzed interviews using thematic (qualitative) analysis and textual (quantitative) analysis, and used the resulting themes to generate items for two social validity questionnaires. We examined the inter‐rater reliability of the questionnaire development process and evaluated the content validity of the questionnaires resulting from each method. Textual analysis had higher inter‐rater reliability for producing themes that could be converted to questionnaire items. The textual analysis method produced a questionnaire with content validity equal to that of the thematic analysis method. The study demonstrates the successful use of a quantitative approach to the development of social validity questionnaires for behavioral interventions.
Behavioral Interventions, 2022 · doi:10.1002/bin.1832