Site icon ICE Blog

Validity (ugh) & Assessing Learners: It’s more than psychometrics

By Jonathan Sherbino (@sherbino)

In a previous post, I suggested that a key element in designing an assessment program is ensuring that a judgment about a learner’s global competence are valid.

Ok, come back. Please. I know I used the “v” word, which might have caused this in your brain…

But the concept of validity is one that a CE must understand.  Importantly, the traditional psychometric approach to validity is changing to parallel changes in the organizing frameworks of medical education.

From Educational Design: A CanMEDS Guide for the Health Professions

Samuel Messick, proposes a contemporary understanding of validity that moves past the statistical accuracy of quantitative scores. “Validity is the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores.”
Kane suggests that validity is better understood as a series of arguments that support the ultimate judgment of global competence of a learner. (From Kane M. An argument-based approach to validation. Psych Bull 1992;112:527-35.)

Essentially, there are 4 key steps in building the argument:

1. Scoring (from observation to score)

2. Generalization (from single score to global score)

3. Extrapolation (from judgment to determination of competence)

4. Decision

(For a very nice review of this argument-based approach, see Holmboe & Hawkins Practical Guide to the Evaluation of Clinical Competence ).

The punch line is this: validity is a web of inferences. Validity is not a “number,” rather it is an argument that supports the final judgment as being “true.”

 Image by NY used under Creative Commons 3 License 

Diagram courtesy of Educational Design: A CanMEDS Guide for the Health Professions 

Exit mobile version