The Universal Translator for Medical Notes
Imagine a doctor in Minnesota and a doctor in Kentucky both writing a note about a patient with a cough. Even though they are talking about the same thing, they might use totally different words, shorthand, or even different computer programs to save that note.
For a computer trying to learn from these notes, it feels like it is trying to read five different languages at once. This messiness is caused by the heterogeneity of ETL processes—which is basically like trying to plug a square peg into a round hole when moving data from one computer to another.
Historically, scientists taught AI to read medical notes at just one hospital. But when they tried to use that same AI at a different hospital, the computer got confused and failed. It didn't understand the local "slang" of the new doctors.
Building a Better Listener
Now, a team of researchers has built a new "universal translator" framework. They used 369 clinical notes from the Mayo Clinic, the University of Kentucky (UKen), and the University of Minnesota (UMN) to see if they could teach an AI named MedTagger to be a better listener.
Sijia
Liu
The results of our use case test run inform us of the importance of a multi-site federated development, evaluation, and implementation framework.
The Training Methods: An Accent vs. A Tour
Single-site Ruleset
This is like teaching a robot to speak only with a Minnesota accent. The AI is trained exclusively on data from one location.
Multi-site Ruleset
This is like giving the robot a tour of the whole country so it can hear many different voices and documentations styles.
The Data Shows a Winner
Single-Site Training
The AI trained only in Minnesota achieved an F1-score of 0.694 at the Kentucky hospital.
Multi-Site Training
Using the multi-site training method, the AI's score at Kentucky jumped to 0.806—a significant improvement.
Peak Performance
At the Mayo Clinic, the multi-site rules reached an even higher performance, achieving an F1-score of 0.884.
The "Sarcasm Detector" for Notes
The scientists also used something called the ConText Algorithm. This acts like a ==sarcasm detector== for the computer. It helps the AI realize that if a note says "the patient does NOT have a fever," the patient is actually healthy, rather than flagging the word "fever" as a problem.
The Limits and Mysteries
The Brochure Problem
The AI sometimes got tricked by "patient education" flyers. It would see a list of COVID-19 symptoms on an instruction sheet and think the patient was sick, even if they were just being handed a brochure!
The Human Mystery
In Kentucky, the Inter-annotator agreement—a measure of how much two people agree on what they are reading—was only 0.211. This reveals that doctors in different states have very different "documentation cultures," or habits of how they write.
Key Takeaway: The team had only small datasets from the outside hospitals (20 notes from UKen and 36 from UMN), indicating they need more data to refine the AI further. For now, they’ve proven a powerful concept: when hospitals work together, the AI learns much faster than when it studies alone.
Reference: An Open Natural Language Processing (NLP) Framework for EHR-based Clinical Research: A Case Demonstration Using the National COVID Cohort Collaborative (N3C). Sijia Liu, Andrew Wen, Liwei Wang, et al.