It may not be worth the effort! Trained judges' global ratings as a criterion measure of social skills and anxiety.

Wallander et al. (1983) asked a simple question. Does training judges help them rate social skills better?

They compared trained judges with untrained college peers. Everyone watched the same short videos of people talking.

The judges gave global ratings of social skill and anxiety. The team then checked how much the raters agreed.

Training gave only a tiny boost in agreement. Trained and untrained judges scored almost the same.

Both groups produced ratings that looked real-world valid. In short, the costly course did little.

Segal (1987) seems to disagree. That study found computer training pushed observer accuracy above 90%. The key difference is task type. F taught moment-by-moment coding with instant feedback. L et al. used broad global ratings with no feedback.

Matson et al. (2004) lines up with the target paper. Caregivers using a motivation profile also showed only moderate agreement, showing the pattern holds across tools and populations.

Singh et al. (1993) backs the skepticism. That paper warns against leaning on broad trait ratings, just as L et al. warn against over-training raters for global scores.

Stop pouring hours into judge boot camps. For everyday social-skills checks, a quick rubric and any willing rater will do. Save your budget for teaching the learner, not the observer.