A minor error in Google’s Med-Gemini model, which merged the names of two different brain regions, highlights the significant risks of artificial intelligence in medicine. Experts call it ‘extremely dangerous’ and warn against ‘automation bias,’ the human tendency to overtrust automated systems.
Article Contents:
- The “basilar ganglia” error: when AI invents anatomy
- Google’s response: a “typo” or a dangerous hallucination?
- Why is this error “extremely dangerous”?
- Automation bias: the greater danger is not machine error, but human trust
- Ongoing errors: not the first incident
- What do experts say?
- Conclusion: a call for cautious skepticism in the era of smart medicine
- Frequently asked questions
The “basilar ganglia” error: when AI invents anatomy
Imagine your doctor uses an AI model to read your brain scan. The model says you have an issue in the “basilar ganglia,” a term that merges two completely different areas of the brain: the “basal ganglia” and the “basilar artery.” This is not just a minor mistake; it is the invention of a non-existent part of the brain. This is precisely the real error made by Google’s healthcare AI model, Med-Gemini, in its own research paper.
Google’s response: a “typo” or a dangerous hallucination?
Worse than the error itself is how Google handled it. When neurologist Brian Moore pointed out the mistake, the company quietly edited the blog post to correct the error without any public acknowledgment, while the original research paper remained unchanged. Google initially described the incident as merely a “typo.” However, following public pressure, the company restored the incorrect result with an explanatory note, stating that “basilar” is a “common transcription error” for “basal” that the model learned from training data. Many medical professionals view this minimization of the error as dangerous.
Why is this error “extremely dangerous”?
Dr. Molin Shah, chief medical information officer at Providence, says: “What you’re talking about is extremely dangerous. Two letters, but a big deal.” A stroke or any defect in the basal ganglia is treated completely differently than an issue in the basilar artery. Confusing the two can lead to a misdiagnosis and an incorrect treatment plan, with potentially severe consequences for the patient.
Automation bias: the greater danger is not machine error, but human trust
Perhaps the greater danger is not that systems sometimes make mistakes, but how credible and trustworthy they seem when they tell you something wrong. This sparks a well-documented phenomenon known as “automation bias,” where complacency causes people to miss errors in a system that is correct most of the time. Dr. Shah says: “The problem with these typos or other hallucinations is that I don’t trust our humans to review it.” If doctors grow accustomed to AI accuracy, they may stop checking its output, and when the AI makes a mistake, it could pass unnoticed.
Ongoing errors: not the first incident
This was not the first time. In newer models like MedGemma, medical professionals found that if they phrasing questions differently when asking the AI, the answers vary and can lead to inaccurate outputs. In one example, the model correctly diagnosed an issue on an X-ray when given a detailed prompt, but stated that the X-ray was “normal” when given a simpler prompt.
What do experts say?
Dr. Jonathan Chen, assistant professor at Stanford University School of Medicine, says: “It’s a very strange threshold moment where a lot of this stuff is being adopted very quickly into clinical care. It’s really not mature.” Dr. Judy Gichoya from Emory University adds: “Its nature is that it tends to make things up, and it doesn’t say ‘I don’t know,’ which is a very big problem in high-stakes fields like medicine.”
Conclusion: a call for cautious skepticism in the era of smart medicine
While some parts of the healthcare industry are in urgent need of help from AI tools, the industry needs to adopt appropriate skepticism before adopting them. The issue is not just that AI makes mistakes—humans make them too—but that AI mistakes can spread and compound rapidly, and our overconfidence in technology can make us ignore these errors. Dr. Shah says: “In my opinion, AI should have a much higher error threshold than a human. Otherwise, I’ll just keep my human to do the work.”
Frequently asked questions
What is the Med-Gemini model?
It is a suite of AI models from Google designed for the healthcare sector. They can summarize health data, generate radiology reports, analyze electronic health records, and more.
What is “automation bias”?
It is the human tendency to rely excessively on automated systems and trust their outputs, which can lead to missing errors when these systems make them.
Is AI ready for use in hospitals?
Many experts believe the technology is still not mature enough for use in high-stakes clinical settings without strict human oversight and careful auditing. It can be useful as an assistive tool, but not as an independent decision-maker.
