CBME, by the ICBME Collaborators, Competency-based medical education
by ICE Blog Admin

Summative Entrustment in Undergraduate Medical Education: Should we?  Can we?  How do we?   

By: Michael S. Ryan (https://www.linkedin.com/in/michael-s-ryan/)

“The post-graduate school was originally an undergraduate repair shop. Urgency required that in the shortest possible time the young physician already involved in responsibility should acquire the practical technique which the medical school had failed to impart.”1 – Abraham Flexner, 1910

Should we make summative entrustment decisions in UME?   

For generations, medical schools have made an implicit statement that its graduates are ready for indirect supervision. When we provide the degrees of MD, DO, MBBS, or MBBCh (in combination with licensing examinations), we afford physicians-in-training with the ability to write orders on patients, to consent them for procedures, and many other of critical tasks required of physicians. Yet, as the quotation from Abraham Flexner above highlights, we have struggled with the trustworthiness of these implicit decisions since the birth of our profession. And though we have made strides to improve upon our graduates’ readiness for internship, there remains a substantial gap.1  

The CBME community has therefore generated substantial interest around formalizing and making explicit the process to render summative entrustment decisions both in graduate medical education (GME) and more recently, in undergraduate medical education (UME).  So, the question of whether we should or should not make entrustment decisions in UME is probably the wrong question. We already make entrustment decisions in UME.  The better question is: can make these decisions in a more explicit, formal, and evidence-based manner?  

Can we make summative entrustment decisions in UME?   

Through their work associated with the AAMC’s Core EPA pilot,2 David Brown and colleagues have demonstrated that, yes, we can make summative entrustment decisions in UME.3,4  At seven of the ten institutions in the pilot, formal summative decisions were rendered.5 A similar experience has been demonstrated at the University of Virginia, where an entrustment committee has and continues to meet three times a year to make entrustment decisions on all students in the program.6

How can we make summative entrustment decisions in UME that are trustworthy?  

Hopefully it’s now clear that we have both an obligation to make entrustment decisions and that executing such decisions is possible.  That leaves us with one final critical question: how do we feel about the trustworthiness of making these formal decisions?    The short answer is: we aren’t quite sure.  Even though many of the medical schools in the Core EPA pilot made entrustment decisions, it is worth noting that these decisions were made entirely for investigational purposes. They did not have a consequence for the learners in the programs. In fact, student leaders from the schools advocated against using entrustment for high-stakes decisions commenting that, “entrustment decisions should not be tied to high-stakes decisions such as graduation until the process has been fully piloted and sufficient instruments have been developed to conduct valid and reliable assessments.”7  

So, what are the challenges we must overcome to make summative entrustment decisions that are trustworthy in UME?  Below is a list that is not exhaustive, but represents a starting point:   

  1. External requirement: Medical schools have no mandate to make entrustment (or competency, for that matter) decisions. For example, the World Federation for Medical Education (WFME) requires that curricular outcomes are organized around the mission of the program and are aligned with objectives and curricular content,8  while US and Canadian programs require only that schools develop program objectives in “competency-based terms.”9,10 External mandates, often through accrediting bodies, have been highly successful in ensuring competency-based decisions are made in the GME environment,11 and a similar commitment may be needed to move the needle in UME.   
  • Valid and reliable data points from multiple sources: Entrustment decisions require a programmatic assessment approach, which in turn requires numerous, multi-source, valid and reliable data points.12 To date, the validity of EPA-based data has been mixed and almost exclusively limited to direct observations in the workplace13–15
  •  Management of data: Medical students are a relatively challenging group to assess. Most medical schools around the world include cohort sizes of 100 or more, with the bulk of clinical experience occurring toward the conclusion of training. Therefore, management of data requires substantial infrastructure to collate data points and efficient review to render decisions in a relatively short timeframe. 
  •  Longitudinal experiences with increased responsibility: Historically, medical schools have oriented clinical training around a clerkship model where students rotate through various specialties. While beneficial in helping students choose specialties, the rotational structure poses substantial difficult in assessing a student’s developmental trajectory. Longitudinal integrated clerkships (LICs) have evolved to counter this impact to great success,16 however, these models are still in the minority across medical schools.  
  • Residency selection: A “successful match” in residency represents the implied goal of medical school education, a reality that hinders advancement of CBME.17 When selection is removed from the equation,18 learners are able to focus on growth-mindset.  With the exception of small pilots, however, the UME to GME transition remains discontinuous resulting in a focus on advancing to the next step. 

 Conclusions 

In summary, there is substantial momentum to apply a summative entrustment decision in the UME context.  It is critical to our commitment to our patients and is possible as demonstrated by several institutions that have piloted the process. However, the remaining challenge involves rendering summative decisions in a way that facilitates trust among our constituents.  Hopefully, work in this area will continue to advance, and we will become more capable of translating our implicit decisions to a more explicit and trustworthy determination.

About the authors:
Michael S. Ryan, MD, MEHP is the Associate Dean for Assessment, Evaluation, Research and Innovation, the Director for the Center for Medical Education Research and Scholarly Innovation (CMERSI) and the Professor of Pediatrics in the Division of General Pediatrics as a hospitalist at the University of Virginia. Dr. Ryan has most recently been on the faculty at Virginia Commonwealth University where he was the Barry V. Kirkpatrick, MD, Vice Chair of Education and Associate Program Director in the Department of Pediatrics and the Assistant Dean for Clinical Medical Education.

References:

  1. Lyss-Lerman P, Teherani A, Aagaard E, Loeser H, Cooke M, Harper GM. What training is needed in the fourth year of medical school? Views of residency program directors. Acad Med J Assoc Am Med Coll. 2009;84(7):823-829. doi:10.1097/ACM.0b013e3181a82426
  2. Lomis K, Amiel JM, Ryan MS, et al. Implementing an Entrustable Professional Activities Framework in Undergraduate Medical Education: Early Lessons From the AAMC Core Entrustable Professional Activities for Entering Residency Pilot. Acad Med J Assoc Am Med Coll. 2017;92(6):765-770. doi:10.1097/ACM.0000000000001543
  3. Brown DR, Moeller JJ, Grbic D, et al. Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study. Acad Med J Assoc Am Med Coll. 2022;97(4):536-543. doi:10.1097/ACM.0000000000004242
  4. Brown DR, Moeller JJ, Grbic D, et al. Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020. JAMA Netw Open. 2022;5(9):e2233342. doi:10.1001/jamanetworkopen.2022.33342
  5. Amiel JM, Andriole DA, Biskobing DM, et al. Revisiting the Core Entrustable Professional Activities for Entering Residency. Acad Med J Assoc Am Med Coll. 2021;96(7S):S14-S21. doi:10.1097/ACM.0000000000004088
  6. Keeley MG, Bray MJ, Bradley EB, Peterson CM, Waggoner-Fountain LA, Gusic ME. Fidelity to Best Practices in EPA Implementation: Outcomes Supporting Use of the Core Components Framework From the University of Virginia Entrustable Professional Activity Program. Acad Med J Assoc Am Med Coll. Published online August 16, 2022. doi:10.1097/ACM.0000000000004944
  7. Geraghty JR, Ocampo RG, Liang S, et al. Medical Students’ Views on Implementing the Core EPAs: Recommendations From Student Leaders at the Core EPAs Pilot Institutions. Acad Med. 2021;96(2):193. doi:10.1097/ACM.0000000000003793
  8. World federation for medical education. Basic medical education WFME global standards for quality improvement: the 2020 revision. Published 2020. Accessed May 24, 2023. https://wfme.org/wp-content/uploads/2020/12/WFME-BME-Standards-2020.pdf
  9. Committee on Accreditation of Canadian Medical Schools. Standards for accreditation of medical education programs leading to the MD degree. Published February 2023. Accessed May 23, 2023. https://cacms-cafmc.ca/wp-content/uploads/2023/03/CACMS-Standards-and-Elements-AY-2024-2025.pdf
  10. Standards, Publications, & Notification Forms. LCME. Accessed September 30, 2022. https://lcme.org/publications/
  11. Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next GME Accreditation System — Rationale and Benefits. N Engl J Med. 2012;366(11):1051-1056. doi:10.1056/NEJMsr1200117
  12. Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Med Teach. 2015;37(11):983-1002. doi:10.3109/0142159X.2015.1060308
  13. Ryan MS, Khamishon R, Richards A, Perera R, Garber A, Santen SA. A Question of Scale? Comparison of Generalizability in Ottawa and Chen Scales When Used to Formulate Ad Hoc Entrustment Decisions for the Core EPAs. Acad Med J Assoc Am Med Coll. 2021;96(11S):S214-S215. doi:10.1097/ACM.0000000000004282
  14. Holmstrom AL, Chia MC, O’Brien CL, Odell DD, Burke J, Halverson AL. Entrustable Professional Activity-Based Summative Performance Assessment in the Surgery Clerkship. J Surg Educ. 2021;78(4):1144-1150. doi:10.1016/j.jsurg.2020.12.001
  15. Cutrer WB, Russell RG, Davidson M, Lomis KD. Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales. Med Teach. 2020;42(3):325-332. doi:10.1080/0142159X.2019.1686135 1
  16. Bonnie LHA, Cremers GR, Nasori M, Kramer AWM, van Dijk N. Longitudinal training models for entrusting students with independent patient care?: A systematic review. Med Educ. 2022;56(2):159-169. doi:10.1111/medu.14607
  17. Ryan MS, Lomis KD, Deiorio NM, Cutrer WB, Pusic MV, Caretta-Weyer HA. Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School. Acad Med J Assoc Am Med Coll. Published online March 24, 2023. doi:10.1097/ACM.0000000000005220
  18. Murray KE, Lane JL, Carraccio C, et al. Crossing the Gap: Using Competency-Based Assessment to Determine Whether Learners Are Ready for the Undergraduate-to-Graduate Transition. Acad Med J Assoc Am Med Coll. 2019;94(3):338-345. doi:10.1097/ACM.0000000000002535

The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The University of Ottawa. For more details on our site disclaimers, please see our ‘About’ page