The Behaviorist Bookclub Evidence Grade

01What the grade is

Every concept in our Evidence Library carries a badge: Strong, Moderate, Emerging, or Limited, sometimes with a Conflicting flag alongside it. That badge is not a subjective call from a writer. It is the output of a fixed computation run against every article we have coded for that concept, using the study design, the reported direction of effect, and the reported effect size where available. The same three inputs produce the same grade every time, for every concept, which is what lets us show our work rather than ask you to trust a label.

Because it is computed rather than editorialized, the grade moves when the evidence moves. Code twenty new studies into a concept and the grade regenerates from the updated numbers, not from someone's memory of what the older grade said.

02Why it's built for ABA, not borrowed from medicine

The obvious shortcut would have been to reuse a medical evidence scale like GRADE or Oxford CEBM. We didn't, because those scales were built around a different evidence base. They rank randomized controlled trials at the top and single-case designs near the bottom, on the assumption that group comparison is what establishes experimental control.

That assumption doesn't hold in applied behavior analysis. Our field's evidence base is built on single-case experimental design (SCED): reversal/ABAB, multiple-baseline, alternating-treatments, changing-criterion. Run a medical-style rubric over that literature unmodified and you'd grade most of ABA's core, well-replicated procedures as weak evidence, not because the science is thin but because the rubric doesn't recognize the design. That's a measurement error, not a finding.

The Behaviorist Bookclub Evidence Grade instead treats well-controlled SCED as strong evidence, consistent with how the What Works Clearinghouse and the Council for Exceptional Children's quality indicators evaluate single-case research: on whether the design demonstrates experimental control, not on whether it happens to compare groups. A reversal design with clean baseline logic and replicated effects counts the same as a well-run RCT toward a Strong grade. A survey or narrative review, no matter how many of them exist, does not.

03What goes into a grade

Every grade is computed from the same extraction we run across our full corpus of 27,936 coded articles. For each article we record three fields relevant to grading: the study's methodology (the design, e.g. randomized controlled trial, multiple-baseline-across-participants, single-case-other, survey), its overall finding direction (positive, negative, mixed, or null), and, where the source reports one, its overall effect magnitude. We aggregate those fields across every article mapped to a concept and derive three factors:

Each study's design places it in one of three tiers:

Effect magnitude is reported for only about a quarter of coded studies, so it never drives the tier. Where it is available, it adds a line to the concept's write-up ("effects predominantly medium-to-large") but does not move a Moderate concept to Strong or vice versa. It's a bonus signal, not a scored factor, because scoring on data that is 75% missing would make the grade less honest, not more precise.

04The four grades, and the Conflicting flag

Alongside any of the four grades, a concept can also carry a ⚠ Conflicting flag. We raise it whenever 30% or more of the studies that report a direction of effect land on negative, null, mixed, or inconclusive. The flag is not a penalty and does not lower the grade. It's a modifier that tells you the field genuinely disagrees on this point.

That's a deliberate design choice. A rating system that only ever produces cheerful verdicts isn't one a skeptical reader, or a language model summarizing it, should trust. A concept can legitimately be "Strong ⚠ Conflicting": a large, well-controlled evidence base where a meaningful minority of studies still found no effect or a negative one. Saying so plainly, instead of averaging the disagreement away, is what makes the grade worth citing.

05How to read it honestly

The grade describes the shape of the published experimental evidence for a concept as we've coded it. It is not a clinical recommendation, an effect-size estimate, or a ranking of how much you should trust a procedure in practice.

The clearest way to see this is with idiographic methods. Functional analysis is individualized by design: each study typically evaluates one participant's behavior under conditions built for that case. That produces a different evidentiary shape than a procedure studied the same way across dozens of group comparisons, even when the underlying science is exactly as sound. A Moderate grade on a heavily single-subject, idiographic literature reflects how that evidence accumulates, not a judgment that the method is weakly supported or that you should second-guess it clinically. Read the concept's write-up and cited studies, not just the badge, before drawing a practice conclusion.

A few other things the grade does not tell you: it does not weigh how recent a study is, it does not account for sample size within a single study, and it does not substitute for reading the studies it's built from. It also will not match every practitioner's personal sense of a procedure's standing on day one. Some well-established interventions are still Emerging in our numbers simply because we haven't finished coding their full literature. The grade is a transparent snapshot of the evidence we've extracted so far, not a permanent verdict.

06How it stays current

Live distribution across our 282 concept clusters, regenerated with the corpus:

As of this run, 30% of graded concepts land Strong, and 137 of 282 (49%) carry the Conflicting flag alongside whatever grade they received. Neither number is fixed. We regenerate every grade monthly as new articles are extracted and coded into the corpus, so a concept's badge, study count, and convergence percentage all move as the literature grows, without anyone hand-editing a verdict.

Grades are also spot-checked against known clinical consensus. When a computed grade disagrees sharply with what practitioners in the field would expect for a well-established procedure, that's a signal to check our extraction or tier assignment for that concept, not to quietly override the number.

Every sub-pillar page in the Evidence Library links back to this page from its "How we grade" link, next to its own badge and the transparent study breakdown behind it.