Learning Analytics: An Introduction to a Future of Medical Education

By Eric Warm (@CincyIM)

Over the past 30 years, the quantity of digital information has doubled about every 2.5 years, totaling 5 Zettabytes in 2014 (5 X 1021 bytes).  Future planners (brain mappers, space explorers, national security advisors) are creating storage facilities as big as a Yottabyte, or 1024 bytes.  (For comparison, the individual genomes of every person on earth amounts to about 1 X 1019 bytes.)

Medical education is contributing to this data boom.

Concepts such as competency based medical education and programmatic assessment shift us away from collecting relatively few high stakes assessments per year per learner, to collecting multiple low stakes assessment points over time. Programs and learners generate hundreds of thousands of data points every day.

What are we doing with all this information?

Learning Analytics are techniques that ‘interpret educational data to describe, characterize and predict the learning behaviors of individuals in higher education.’

Data analytics exist in sports, business, finance, weather, traffic, and wherever large amounts of data are created and have potential value.

The analytics maturity model has five steps.

  1. Descriptive analytics –  data are used to describe what happened, for example, how a learner performed in a given clinical scenario
  2. Diagnostic analytics – data are used to determine why something happened, for example, why a learner performed the way they did
  3. Predictive analytics – data are used to determine what will happen next, for example, predicting how a learner will do on the next rotation based on their performance to date
  4. Prescriptive analytics – data are used to determine what should be done in the future, for example, determining what it would take to help a learner bend their learning trajectory towards improvement
  5. Cognitive/Self Learning analytics – data are used to determine what we do not know, for example, artificial intelligence or machine learning to identify patterns and connections in a learner’s data that humans cannot see

In this model, analytics moves from a reactive position (what did we see and why did it happen?), to a proactive position (what will happen next?), to optimization and foresight (what should we do? what can’t we see?).

Sounds great, right?

It can be. Learning analytics has the power to provide high quality information to improve individual learning trajectories, advance program performance, and even tie educational assessment to patient care outcomes.

But, there are risks. Learning analytics requires sound data collection, aggregation, analysis, and reporting, but each of these steps is prone to common human foibles such as bias, or action without deep understanding or expertise.

A recent paper led by Brent Thoma details many of the concerns raised about learning analytics: including data security, governance, analysis, access, presentation, validity, and consequences, – and these will need to worked out to achieve the potential benefit.

I’ve titled this post “A Future of Medical Education”, but it really should be called “The Future of Medical Education.” We will be doing Learning Analytics. Humans don’t have the greatest track record of not doing something that is possible. Split an atom? Check. Clone a sheep? Check. Carry around a device in our pocket that tracks our every movement? Check. Each of these endeavors has great positive potential, but also great negative possibilities.

A recent paper published in PLOS ONE detailed a project in which researchers developed an artificial intelligence system that uses radio wave signals (like the Wi-Fi you are likely using right now) to identify heart rate and breathing patterns to detect your emotions (anger, joy, sadness, pleasure). Some have suggested that in the future this technology could help with human robot interaction or detecting the mood of workers when the boss rolls out a new initiative. How does this feel to you? Some good stuff and less good stuff?

Learning Analytics, as a technology of the present and future, is no different.  We need to make sure we maximize the good and minimize the bad.

If you are interested in hearing a debate on the pros and cons of Learning Analytics in medical education, consider watching the presentation called Learning Analytics At The Cutting Edge: Validity And Artificial Intelligence at the International Conference of Residency Education Meeting on May 19th, 2021.

As you learn about Learning Analytics you will find a diverse field filled with many variations, viewpoints, and applications. You might want to begin your journey with some of these works:

General Overviews:

Learning Analytics in Medical Education Assessment: The Past, the Present, and the Future

Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus from an International Cohort of Medical Educators

Developing the role of big data and analytics in health professional education

Using Electronic Health Record Data to Assess Residents’ Clinical Performance in the Workplace: The Good, the Bad, and the Unthinkable

“Yes, and …” Exploring the Future of Learning Analytics in Medical Education

The Time Is Now: Using Graduates’ Practice Data to Drive Medical Education Reform

Specific Applications and Examples:

The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance

Learning Curves in Health Professions Education

How Well Is Each Learner Learning? Validity Investigation of a Learning Curve-Based Assessment Approach for ECG Interpretation

Developing A Dashboard to Meet the Needs of Residents in A Competency-Based Training Program: A Design-Based Research Project

Developing Resident-Sensitive Quality Measures: A Model from Pediatric Emergency Medicine

Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency

Image via Techno FAQ

The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page