By: Deena M. Hamza (@DrDeenaMHamza)
Leading the evaluation of competency-based medical education (CBME) at the University of Alberta, Canada, has taught me more about innovation than any book I have read on the topic. Introducing change is a feat even in optimal conditions where all layers of an organization’s systems are invested in new ways of doing things, and change is co-created with innovators, providers and recipients of the change, and regulatory and governing bodies. Consider the challenges when innovation is prescribed and mandated – contentious, right? Now, add uncertainty about the actual outcomes and impact of the changes brought on by innovation, and we have the conditions that breed tension and resistance. We’ve all heard the comments from front-line implementers:
- “This requires so much of my time and effort, are you sure this [insert innovation] works?”
- “Why should we change the ways things are with [insert innovation]. Things are fine the way they are.”
- “This is a political move. {Insert innovation} will not change anything worth changing.”
- “This [insert innovation] doesn’t work for our unique context. We’re different.
In leading a BEME Scoping Review on CBME Literary Conversations, uncertainty and complexity were two recurrent themes I found to be prominent. Uncertainty about the outcomes and impact of CBME as a curricular innovation, as well as how curricular innovations more broadly may (or may not) influence practice and subsequent patient care outcomes. Conversations about uncertainty were almost always tied to questions about if CBME works, which is more complex than what a yes/no dichotomy can provide. A fundamental issue that has yet to be confronted is what evidence is needed as proof that CBME works? Is there a singular answer determined by consensus? Will this be different by context and/or interest group? Is the answer static or dynamic, evolving over time? See – it’s complex!
In October 2023, I shared some ideas when I was an invited speaker at a program evaluation workshop at the International Conference on Residency Education, hosted in Halifax, Nova Scotia, Canada. To my surprise, participants were nodding along as I critiqued current evaluation methods that require one to be objective, value-free, and prepared to pass judgement about the merit, value, and worth of CBME with a snapshot of evidence collected in discrete moments in time. In this presentation I challenged the group to think differently about evaluation, specifically to see Developmental Evaluation, created by Dr. Michael Quinn Patton, as the way forward. Most of the time, those conducting evaluation of CBME are individuals embedded in the change, so we are in fact value-laden and not objective – and, I suggested that we need to embrace and nurture this rather than reject it on the basis of a paradigmatic hierarchy that suggests subjectivity is less credible and trustworthy. We are also working in spaces of uncertainty and emergence; we are learning as we go. So, a priori metrics and key performance indicators are not amenable to contexts where we need to try new practices and activities, reflect on how they play out in different contexts, and course correct in real-time. These are all the central tenets of Developmental Evaluation – capturing rich stories of change during uncertainty and complexity to inform next wise actions.
For over a year, I have had the privilege of working closely with Dr. Patton as I lead the world’s largest Developmental Evaluation of team-based training and education innovations across Canada. I have learned how to lead a team of evaluators to embrace the principles of this evaluation philosophy, and the practical methods with which we collect evidence that conveys rich and robust stories about change that are as diverse as the contexts within which we work. Each evaluation is unique; each story contributing to the evidence of worthwhile change.
As mentioned in the BEME Scoping Review, evaluation studies about CBME are still in their infancy; more than two-thirds of the included articles were perspectives/commentaries. As a collaborative, there is much work to be done to 1) conceptualize what constitutes evidence (and for whom); and 2) what methods are appropriate to capture evidence in both uncertain and complex contexts. In doing so, I believe a working framework of what evidence is needed and how to collect it will result in rich stories about the implementation of CBME, globally, and if/how this curricular innovation works (or not) in diverse contexts.
About the author: Deena M. Hamza, PhD, is an Innovation & Health Professions Education Scientist, Director of Research & Evaluation for Postgraduate Medical Education, and Adjunct Assistant Professor in the Department of Medicine at the University of Alberta. She is also the Chair of the Canadian Association for Medical Education Foundation. Dr. Hamza’s program of research centers on how to transform systems and the longevity of innovations using her recently published framework called Eco-Normalization.
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The University of Ottawa . For more details on our site disclaimers, please see our ‘About’ page