By Jonathan Sherbino (@sherbino)
In a previous post, I suggested that a key element in designing an assessment program is ensuring that a judgment about a learner’s global competence are valid.
Ok, come back. Please. I know I used the “v” word, which might have caused this in your brain…
But the concept of validity is one that a CE must understand. Importantly, the traditional psychometric approach to validity is changing to parallel changes in the organizing frameworks of medical education.
Samuel Messick, proposes a contemporary understanding of validity that moves past the statistical accuracy of quantitative scores. “Validity is the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores.”
Kane suggests that validity is better understood as a series of arguments that support the ultimate judgment of global competence of a learner. (From Kane M. An argument-based approach to validation. Psych Bull 1992;112:527-35.)
Essentially, there are 4 key steps in building the argument:
1. Scoring (from observation to score)
- How were the instruments constructed?
- Were the scoring algorithms applied correctly?
- How is qualitative data reported?
- Is it feasible?
2. Generalization (from single score to global score)
- What are the sources of measurement error?
- Can the scores be replicated?
- Is the assessment process constructed in a systematic way?
3. Extrapolation (from judgment to determination of competence)
- What is the process of member checking and collation?
- Do scores predict real world performance?
- Does the artificial nature of testing impact the scores?
4. Decision
- Can the standard be defended?
- If a candidate fails the standard, will remediation lead to improvement in a future score?
(For a very nice review of this argument-based approach, see Holmboe & Hawkins Practical Guide to the Evaluation of Clinical Competence ).
The punch line is this: validity is a web of inferences. Validity is not a “number,” rather it is an argument that supports the final judgment as being “true.”
Image by NY used under Creative Commons 3 License
Diagram courtesy of Educational Design: A CanMEDS Guide for the Health Professions