Low interrater agreement on the semantic base of textual material.

The team ran three small experiments. In each one, they asked different raters to read short texts.

The raters had to pick out the “semantic base” — the hidden meaning Skinner said controls what we say.

Across all tests, the raters rarely agreed. Agreement scores stayed below the level we accept for good data.

Even trained people could not agree on what a text “really means.”

Because raters disagreed, the idea of a stable “semantic base” failed a basic reliability check.

The authors warn: if we can’t measure it reliably, we can’t use it in science.

Baer et al. (1984) already showed that most papers only talk about Skinner’s verbal operants; few test them. L et al. now give one reason why — the key unit can’t be rated reliably.

Lancioni et al. (2008) later found the same trouble with visual inspection of graphs. Low agreement keeps popping up, no matter what we ask raters to judge.

Lerman et al. (1995) looks like a contradiction — they got 95 % agreement on a sex-ed interview. The difference is simple: they used clear, scripted questions, not open “what does this mean?” judgments. Good agreement is possible when the item is concrete.

If you write a goal that hinges on identifying “meaning,” you are building on sand. Use observable verbal operants — mand, tact, intraverbal — that two people can count the same way. Pick targets you can hear and see, not ones you have to interpret.