Site icon ICE Blog

Programmatic Assessment: Moving Beyond Psychometrics Not Away From

ICE Blog Admin

By: Daniel J. Schumacher, MD, PhD, MEd

This image was generated by ChatGPT 4o based on the content of this blog

In 2013, Brian Hodges penned a seminal piece entitled, “Assessment in the post-psychometric era: learning to love the subjective and collective.”1 In the years since, several articles have underscored the value of narrative assessment data and judgments made by humans such as those who sit on clinical competency committees (CCCs).2-4 This has led educators to gain the “insight that subjective expert judgments by medical professionals are not only unavoidable but actually should be embraced as the core of assessment of medical trainees” because the perception of objectivity in assessment is a fallacy.5 While valuing the subjective and collective is now an accepted component of best practice in assessment, I am not sure everyone agrees with that. The siren call of solely psychometrics is hard for many to resist. If this might be you, I am hoping you will give me a chance to try to change your mind. I will start here – consider this… in Homer’s Odyssey, the alluring songs of the sirens lead passing sailors to their death rather than to a more enlightened state.

Clearly, I have embraced the post-psychometric era of assessment.1 However, I also agree with Jacob Pearce that this era should not be mistaken as an era that moves away from psychometrics or is against psychometrics.6 Moving beyond is not the same as moving away or moving against. The post-psychometric era invites us to take a bigger picture view that is advocated by the principles of programmatic assessment. But, quantitative data should, indeed must in my view, be used in programs of assessment. In the United States, the one approach certifying bodies across all medical and surgical specialties have in common is a traditional secure exam (https://www.abms.org/wp-content/uploads/2023/11/abms-member-board-requirements-for-initial-certification-in-a-specialty.pdf). I believe these traditional secure exams are the gold standard for assessing knowledge and clinical reasoning and are a core component of defensible validity arguments for initial certification decisions.

Why Move Beyond Only Traditional Secure Exams?

While excellent for assessing medical knowledge and clinical reasoning, traditional secure exams leave foundational competencies (e.g., communication, trustworthiness, professionalism, equitable care, collaboration with patients and other healthcare team members, ability to work in systems) important to caring for patients insufficiently addressed. Exam content can include focus in some areas, such as professionalism and ethics, but they cannot sufficiently cover these areas. Thus, these 21st century competencies must be assessed in other ways. I believe these are best assessed in the workplace as part of a programmatic assessment approach that includes ensuring a mix of assessment methods; triangulating assessment information across data points towards a competency framework; and making high stakes decisions in a credible and transparent manner using a holistic approach, perhaps by a CCC.7 Of these, a meaningful triangulation of data to make decisions is core to programmatic assessment.7,8 Bearing this in mind, consider what Pearce and Prideaux wrote regarding the hegemony of examinations:

“Part of the problem in achieving triangulation of data is that data derived from examinations are perceived to be of greater credibility than those derived from other forms of assessment. This is not surprising given that colleges, rightfully, have given much attention to improving the psychometric properties of examinations. There is much less of a history with newer forms of work-based assessment. Although quality considerations are important across a whole programme of assessment, in a programmatic approach it is the patterns across all assessments that are essential. That is not to say that examinations should not carry a lot of weight. Think of a family photograph album. Examinations are like studio portraits: high quality and clear but they cannot tell the whole picture of family life on their own. Attention needs to be given to: how examinations are used and weighted in the triangulation of data across formats; how they are being used to provide feedback to trainees; how they promote learning; and ultimately how they contribute to the making of progression decisions.”

A programmatic approach to assessment highlights several polarities in contemporary assessment that must be managed from the standpoint of both/and rather than either/or9 – all of these require moving beyond psychometrics:

“I’m Sorry, Secret Siren, But I’m Blocking Out Your Calls” (Into the Unknown, Frozen 2)

Like Odysseus steering past the sirens, we too must recognize the danger in yielding to the call of psychometrics alone. This is especially true when making higher stakes decisions, such as board certification, where traditional exams still reign supreme. To be defensible, high stakes decisions should be made using multiple types and sources of data to provide the best picture of an individual’s readiness, consistent with contemporary views of programmatic assessment. One source, or even one dominant source, of data is not defensible. In the parlance of chest x-rays in radiology, “one view is no views.” But, moving to programmatic assessment cannot limit new assessment information to quantitative data either. It must also include embrace subjectivity and human judgment, which promotes fairness in assessment.10

Are you convinced? I hope so.

About the Author:

Daniel J. Schumacher, MD, Phd, MEd is a tenured professor of pediatrics at Cincinnati Children’s Hospital Medical Center. He is a pediatric emergency medicine physician and medical education research scientist. His research focuses on entrustable professional activities, resident-sensitive quality measures, and other areas of CBME.

References

  1. Hodges B. Assessment in the post-psychometric era: learning to love the subjective and collective. Medical teacher. 2013;35(7):564–568.
  2. Scarff CE. Towards a greater understanding of narrative data on trainee performance. Med Educ. Oct 2019;53(10):962–964. doi:10.1111/medu.13940
  3. Bartels J, Mooney CJ, Stone RT. Numerical versus narrative: A comparison between methods to measure medical student performance during clinical clerkships. Med Teach. Nov 2017;39(11):1154–1158. doi:10.1080/0142159X.2017.1368467
  4. Ginsburg S, van der Vleuten CP, Eva KW. The Hidden Value of Narrative Comments for Assessment: A Quantitative Reliability Analysis of Qualitative Data. Acad Med. Apr 2017;doi:10.1097/ACM.0000000000001669
  5. Ten Cate O, Regehr G. The Power of Subjectivity in the Assessment of Medical Trainees. Acad Med. Mar 2019;94(3):333–337. doi:10.1097/acm.0000000000002495
  6. Pearce J. In defence of constructivist, utility-driven psychometrics for the ‘post-psychometric era’. Med Educ. Feb 2020;54(2):99–102. doi:10.1111/medu.14039
  7. Heeneman S, de Jong LH, Dawson LJ, et al. Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles. Med Teach. Aug 03 2021:1–10. doi:10.1080/0142159X.2021.1957088
  8. Pearce J, Prideaux D. When I say … programmatic assessment in postgraduate medical education. Med Educ. Aug 9000;Epub ahead of printdoi:10.1111/medu.13949
  9. Govaerts MJB, van der Vleuten CPM, Holmboe ES. Managing tensions in assessment: moving beyond either-or thinking. Med Educ. 01 2019;53(1):64–75. doi:10.1111/medu.13656
  10. Valentine N, Durning SJ, Shanahan EM, van der Vleuten C, Schuwirth L. The pursuit of fairness in assessment: Looking beyond the objective. Med Teach. Feb 01 2022:1–7. doi:10.1080/0142159X.2022.2031943

The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The University of Ottawa . For more details on our site disclaimers, please see our ‘About’ page

Exit mobile version