AI is transforming healthcare—but confidence is not the same as accuracy. When AI hallucinates, the risk isn't just misinformation; it can become a patient safety issue. Trust in health AI will depend on transparency, source disclosure, and clear standards. If drugs need evidence labels, shouldn't AI-generated medical advice have them too?
AI doesn't lie the way humans do. It doesn't have a motive. It just... believes its own wrong answer, and says it with a straight face.
No flinch. No "I'm not totally sure." No tell. It hands you the wrong dose with the same calm confidence it uses to get the right one. That's hallucination, and it's the part of AI in healthcare nobody puts on the brochure.
Here's the strange bit: the world is leaning on AI for health answers faster than it's learning to trust it. This KFF poll in March 2026 found about one in three US adults now ask AI for health information, basically tied with social media. And yet, a different survey the very next month found people trusting AI in healthcare less, not more, than they did two years ago. We're reaching for it with one hand and side-eyeing it with the other.
That side-eye is earned. Emergency Care Research Institute (ECRI), the kind of patient-safety body hospitals actually listen to, named AI chatbot misuse the single biggest health technology hazard going into 2026. Not faulty machines. Not data leaks. Confidently wrong answers.
And "confidently wrong" has a face. Take methotrexate, a common drug for rheumatoid arthritis. It's dosed weekly, somewhere between 7.5 and 25mg. An AI tool stating "25mg daily" isn't a typo-level slip. It's the gap between treatment and toxicity, said in the exact same tone as everything else it says. A 2025 benchmark testing top AI models on catching this kind of subtle medical falsehood found even the best performer barely scraped past half marks on the hardest cases. A Mount Sinai study fed chatbots slightly wrong medical details just to see what they'd do with it. They didn't catch it. They ran with it, dressed it up, made it sound even more official.
Now bring that home.
Africa already has a trust problem with medicine, and AI didn't cause it. WHO data shows at least 1 in 10 medical products across low- and middle-income countries, plenty of them, are substandard or falsified. In some informal markets across West and Central Africa, that number climbs past 70%. So picture the actual situation: a patient who already can't fully trust the pill, now also can't fully trust the answer telling them how to take it. That's not one risk. That's two risks shaking hands.
A drug can't reach a shelf without disclosing its trial data. Why should an AI tool dispensing medical answers get to skip that conversation entirely? Right now, most do. That's not a tech gap. That's a standards gap. And standards gaps are exactly the kind of thing an industry body should close, before a regulator has to close it the hard way.
The fix isn't "ban the AI," by the way. There's already a promising step in that direction: "Drug Insights," a RAG-based AI chatbot built specifically for African healthcare workers. Instead of answering from general internet training data, it pulls directly from a real corpus of Nigerian pharmaceutical formulary data before it generates a response, meaning the answer is grounded in actual drug information, not guesswork dressed up as confidence. It's still in academic validation and peer review, not yet live in any pharmacy. But early testing, including feedback from practicing pharmacists, found it produced accurate, context-specific answers with far less hallucination than a general-purpose chatbot. Drug Insights proves the direction works. What it can't do alone is set the rule everyone else has to follow.
That rule needs to be a disclosure standard, something as basic as: which formulary, which market, which dosing convention? Call it a formulary label. Every AI tool dispensing medical information should be required to show its sources the way a drug shows its trial data. Simple, checkable, non-negotiable.
A counterfeit pill gets recalled. A counterfeit answer doesn't.
So here's the one I keep circling back to: if we'd never let a drug onto the market without proof of what it was tested on, why have we let AI onto the same shelf without asking the same question.