By: Dr. Daniel Nel (https://www.linkedin.com/in/daniel-nel-264544195/?originalSubdomain=za)
“WBA is a joke, nobody takes it seriously here,” is what a colleague from another country, where it has been used for many years, told me when asked for tips on implementation of WBA.
Programmatic assessment, a core component of competency-based education, relies heavily on WBA. However, an authentic assessment strategy like WBA comes with costs, including an increased assessment and administrative burden on both supervisors and trainees.
If the assessment burden is perceived to outweigh the benefits (especially the educational value), WBA may devolve into a box-ticking exercise. This is an unfortunate situation because it significantly undermines WBA’s value for both summative decision-making and formative feedback for students. In instances where the assessment burden has led to widespread negative perceptions of WBA, there have even been reports of ‘gaming’ the system, with supervisors giving their login credentials to trainees to complete their own observation reports.1
Shifts in assessment thinking
Assessment quotas have largely been informed by psychometric parameters to define adequate reliability for summative decision making. This can result in a hefty assessment burden, depending on the number of activities/EPAs that need to be assessed. However, there is a growing movement to consider WBA beyond the psychometric paradigm, emphasising the value of human judgement in assessments.2 This shift represents a move from a behaviourist to a more constructivist, sociocultural approach to performance assessment, relying more on qualitative aspects rather than numerical metrics.
Setting feasible assessment quotas
With that in mind, the question remains, how can the assessment burden be reduced to the point that trainees and supervisors feel the benefits of participating outweighs the burden? Firstly, you can reduce the number of EPAs by selecting a core or index set that can be reliably observed and infer competence for other tasks that are similar. However, simply selecting an index subset of EPAs presents certain problems, such as trainees potentially only focusing on the index activities and missing out on educational opportunities for activities that are ‘not on the list’. This issue can be mitigated by making EPAs broader and more inclusive rather than narrow and specific. Keeping EPAs broad allows for a range of clinical activities to be included or ‘nested’, preventing trainees from missing valuable learning opportunities for activities not listed. It also enables trainees to request observations on more complex or advanced activities, allowing them to challenge themselves.
Secondly, you can reduce the number of observations required per EPA. Although some have done away with minimum quotas entirely, there may still be a need to set a minimum within a specific time to ensure broad participation. While some ‘early adopters may influence an ‘early majority’ to engage in a new WBA strategy, a significant portion of the team, especially the trainees, may participate only if required. Having some degree of participation is necessary for all trainees to benefit from the formative value of workplace observations with feedback, and to generate enough data points for the Competence Committee (CC) to make high-stakes decisions.
I believe this number should be determined collaboratively with both trainees and supervisors to ensure the assessment burden is acceptable to them. This can be achieved by starting with a very low assessment quota during the initial rollout of the WBA strategy. Once most team members have some experience, they can suggest a reasonable number of assessments that they feel they could manage within a defined period, such as during a clinical rotation/placement. This approach enables the participants to be co-creators of the strategy and take ownership of the assessment requirements, reducing the likelihood of complaints about the feasibility of the requirements or their impact on clinical service demands. For example, in our General surgery team at the University of Cape Town, both supervisors and trainees suggested a mean number of five observations over a three-month rotation period, equating to a minimum of 20 observations per year. It is always possible to gradually increase the quota once the team is accustomed to the system and more invested in it, but setting difficult targets initially is ill-advised.
But what about making reliable high stakes decisions? From a summative perspective, rather than counting whether a trainee has received enough ‘level four’ entrustment ratings for specific EPAs according to a predetermined numerical requirement, the CC members’ wisdom can determine whether adequate data points are present. CC members generally know both the trainees and the supervisors who are conducting the assessments and can subjectively judge whether sufficient information has been presented to make a trustworthy decision regarding a trainee’s competence for a specific EPA. This certainly does not mean that reliability is unimportant; the CC still must ensure adequate sampling. However, whether this has been achieved should depend more on the subjective judgement of the CC, rather than a mathematically calculated number. That being said, a low minimum required number of observations per EPA may still be advisable to provide some guidance, especially for inexperienced teams. For example, our CC requires at least three observations for high stakes decisions (e.g. readiness for progression in training) but acknowledges that many more may be required depending on the trainee.
There are obviously other strategies, beyond setting feasible quotas, that are also important to reduce the assessment burden of WBA. However, the maximum word count for this post is looming and further discussion regarding enhancing the feasibility of WBA must be left for another day…
WBA is indeed a powerful and essential tool. However, it is easy to set quotas that are neither feasible nor sustainable. We must ensure this does not happen and instead strive to make the assessment burden manageable for both trainees and supervisors.
References
- Beamish AJ, Johnston MJ, Harries RL, et al; Council of the Association of Surgeons in Training. Real-world use of workplace-based assessments in surgical training: A UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training. Int J Surg. 2020 Dec;84:212-218
- Schuwirth LWT, van der Vleuten CPM. A history of assessment in medical education. Adv Health Sci Educ Theory Pract. 2020 Dec;25(5):1045-1056
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The University of Ottawa . For more details on our site disclaimers, please see our ‘About’ page