By: Alan Schwartz, PhD, JD.
One of the core components of competency-based health professions education (CBHPE) is sequenced progression: “competencies and their developmental markers are … organized in a way that leads to a logical developmental sequence across the continuum of medical education or practice.”[1] That is, regardless of the competency framework, we should be able to distinguish levels of development and place them in order from lower to higher levels.
Competency developers routinely organize them like this. For example, the CanMEDS milestones identify up to six developmental stages for each CanMEDS 2015 competency[2], and the ACGME milestones identify five developmental stages for each subcompetency.[3] Within these frameworks, assessment focuses on assigning learner performance to competency levels.
How can we know if levels are distinguishable?
Asking whether human beings can distinguish two magnitudes has a long history in psychology and psychophysics. Asking whether two levels of a psychological scale are subjectively different is a natural extension of this history and is broadly called “psychological scaling”. In the case of developmental markers for a single competency, our intuition (and an explicit best practice in competency development) is that the markers represent points along a unidimensional continuum, and increasing competence corresponds to increasing subjective distance from the lowest marker to the higher markers. This is why it is natural to think of sequenced progression as milestones—markers of distance along the path of mastery—serving as a road map for learners and underpinning assessment for learning.[4]
Psychological scaling provides a powerful set of methods for finding out whether this intuition actually holds for a particular set of markers. In a recent paper, colleagues and I employed these methods to look at the five levels of eight U.S. ACGME General Pediatrics milestones.[5] We asked groups of people (residency program directors, competency committee faculty members, and residents in pediatrics and in internal medicine) to consider pairs of pediatric milestones within subcompetencies. We told them which of the milestones was intended to represent the higher level of performance. We asked them how much more competence was represented by the higher vs. lower milestone. We weren’t asking about a particular learner’s competence, but about how the levels of competence were described in the framework.
What we’ve learned
We then applied scaling methods to infer the average subjective distance between each of the five levels of the subcompetency. For some subcompetencies, respondents clearly distinguished the marker levels, with the subjective differences between levels showing a regular increase as expected. But for several subcompetencies, two, three, or even four of the five levels were not reliably perceived as different. Put another way, despite the availability of five levels of markers, users sometimes could really only distinguish four, three, or even just two distinct levels of competence when considering the descriptions of the markers.
By and large, the differences among subcompetencies were equally evident across different user roles and the two subspecialties. But we found some differences in how distinguishable the markers were among different user roles (residents seemed to make fewer distinctions among markers in two subcompetencies (reflective practice and interprofessional/team communication), and users in different specialties distinguished different numbers of levels for some subcompetencies.
Guidance for competency framework developers
These findings have implications for developers of competency frameworks for CBHPE, which often employ expert panels proposing levels and markers that resonate with observations in clinical practice. Rather than assume that our markers reflect a sequenced progression of competence that we can reliably distinguish and assess with a fixed number of levels, we may do well to incorporate explicit tests of distinguishability and sequencing as part of the development cycle. When doing so, we should carefully consider the function of the framework and who is expected to share the developers’ mental models of sequenced progression. We should also consider right-sizing sequences: if a competency only has two or three practically distinguishable markers, its framework should not impose five, six, or nine indistinguishable levels of progression.
Refrences:
- Van Melle E, Frank JR, Holmboe ES, Dagnone D, Stockley D, Sherbino J; International Competency-based Medical Education Collaborators. A Core Components Framework for Evaluating Implementation of Competency-Based Medical Education Programs. Acad Med. 2019 Jul;94(7):1002-1009. doi: 10.1097/ACM.0000000000002743. PMID: 30973365.
- https://canmeds.royalcollege.ca/guide
- https://www.acgme.org/globalassets/milestonesguidebook.pdf
- Schumacher DJ, Lewis KO, Burke AE, Smith ML, Schumacher JB, Pitman MA, Ludwig S, Hicks PJ, Guralnick S, Englander R, Benson B, Carraccio C. The pediatrics milestones: initial evidence for their use as learning road maps for residents. Acad Pediatr. 2013 Jan-Feb;13(1):40-7. doi: 10.1016/j.acap.2012.09.003. Epub 2012 Nov 17. PMID: 23165175.
- Schwartz A, Hobbs K, Schumacher DJ. Scales, Intervals, and Dissonance: Do Interpretations of Harmonized ACGME Milestone Levels Vary by User Role and Specialty?. J Grad Med Educ. 2026;18(1):36-42. doi:10.4300/JGME-D-25-00072.1
About the Authors:
Alan Schwartz, PhD, JD, is a Professor of Epidemiology and Biostatistics and of Paediatrics, Senior Scientist at the Centre for Education Research & Innovation, and Western Research Chair in Medical Education and Medical Decision Making at Western University’s Schulich School of Medicine & Dentistry, London, ON, Canada. (ORCID 0000-0003-3809-6637).
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The University of Ottawa. For more details on our site disclaimers, please see our ‘About’ page