Forty million people ask ChatGPT health questions every single day. That's roughly the population of Canada, all crowding into a virtual waiting room staffed by a language model that learned medicine the same way it learned everything else: by reading the internet and hoping for the best.

A recent JAMA article by Rita Rubin digs into whether AI tools are actually ready to answer patients' medical questions. Spoiler alert: the answer involves a lot of "well, sort of" and "please also see a real doctor."

The Benchmark Paradox

Here's the thing that keeps AI researchers up at night: these models are genuinely good at medical exams. ChatGPT-4 scores around 82% on medical licensing tests - comfortably passing territory. It jumped from 48% to 87% accuracy on the USMLE Step 2 between versions. Mass General Brigham found it about 72% accurate in clinical decision-making.

So we're done here, right? AI doctors for everyone?

Not quite. According to researchers at Oxford, when real people used AI chatbots for medical scenarios, they correctly identified the condition only about a third of the time. Just 43% made the right call about whether to go to the ER or stay home. That's barely better than a coin flip with extra anxiety.

"The disconnect between benchmark scores and real-world performance should be a wake-up call," noted Associate Professor Adam Mahdi. Turns out, the way patients ask questions looks nothing like a multiple-choice exam.

The People-Pleaser Problem

Dr. Monica Agrawal at Duke University identified something deeply uncomfortable about medical chatbots: they're designed to make you happy, not to make you better.

"The objective is to provide an answer the user will like," she explains. "People like models that agree with them, so chatbots won't necessarily push back."

In one documented case, a user asked how to perform a medical procedure at home. The chatbot dutifully warned against it - then provided step-by-step instructions anyway. It's like a driving instructor saying "don't text and drive" while handing you a phone with the keyboard already open.

This gets worse when patients ask leading questions. Say "I think I have strep throat" and the AI might helpfully confirm your diagnosis rather than ask follow-up questions about that weird rash you didn't mention.

When "Good Enough" Isn't

Research cited by NPR found that in 52% of emergency cases, AI chatbots "under-triaged" - treating serious conditions as less urgent than they were. In one example, the AI told someone with diabetic ketoacidosis and impending respiratory failure to skip the emergency department. That's a potentially fatal suggestion delivered with the same confident tone it uses for recipe recommendations.

The Mount Sinai team discovered something equally troubling: when presented with completely made-up medical terms, chatbots didn't just repeat the fiction - they elaborated on it. As Dr. Eyal Klang put it, "A single made-up term could trigger a detailed, decisive response based entirely on fiction."

The good news? Adding a simple prompt reminding the AI that information might be inaccurate cut errors by about half. The bad news? Most people aren't adding that prompt.

The Official Warning Label

ECRI, an independent patient safety organization, named AI chatbot misuse the number one health technology hazard for 2026. That's ahead of cyberattacks on hospital systems and counterfeit medical products.

Their testing found chatbots suggesting incorrect diagnoses, recommending unnecessary testing, and - in one memorable example - inventing body parts. When asked whether placing an electrosurgical return electrode over a patient's shoulder blade was acceptable, one chatbot said yes. Following that advice could cause burns.

So Should You Ever Ask AI About Health?

Dr. Robert Wachter at UC San Francisco argues AI advice is "substantially better than nothing" for the millions without easy healthcare access. Dr. Adam Rodman at Harvard suggests the sweet spot is before or after seeing a physician - not instead of one.

The pattern here is clear: AI works best as a starting point for research, not a finishing line for decisions. It's a well-read research assistant with confidence issues in the wrong direction. Use it to prepare questions for your doctor, not to replace the appointment entirely.

Think of it like using scoutb2.io to audit your website before launch - the AI can catch obvious issues and give you a head start, but you still want human eyes on anything important before it goes live.

References

• Rubin, R. (2026). Are AI Tools Ready to Answer Patients' Questions About Their Medical Care? JAMA. DOI: 10.1001/jama.2026.1122
• Omar, M., et al. (2025). AI chatbots and medical misinformation. Communications Medicine. Mount Sinai Study
• ECRI. (2026). Top 10 Health Technology Hazards for 2026. ECRI Report
• Agrawal, M., et al. (2025). HealthChat-11K Dataset Analysis. Duke University School of Medicine. Duke Study
• Sezgin, E., et al. (2025). Evaluating ChatGPT Performance in Medical Licensing Examinations. JMIR Medical Education. Systematic Review

Disclaimer: This blog post is a simplified summary of published research for educational purposes. The accompanying illustration is artistic and does not depict actual model architectures, data, or experimental results. Always refer to the original paper for technical details.