Evidence Brief · OneRx

Is ChatGPT Reliable for Medical Advice? What the 2026 Evidence Shows for Clinicians

Published 26 July 2026 · Reading time 11 min
By the OneRX Clinical Content Team · Reviewed against source publications prior to release
For healthcare professionals only

On this page: What current models do · The number nobody shows · Where it still breaks · What the fix looks like · Where OneRX differs · What no benchmark measures · What to do on Monday · FAQ · References

If you have sat through a hospital talk on AI safety in the last two years, you have seen the same slide. Thirty-four point three percent. Non-concordant recommendations. JAMA Oncology, 2023.

It is a real finding from a good study. Chen and colleagues at Mass General Brigham ran 104 prompts across 26 diagnosis descriptions in breast, prostate and lung cancer, scored the outputs against NCCN Guidelines, and found that while 98% of responses contained at least one concordant treatment, 34.3% also contained one or more that were not concordant. Another 12.5% included a treatment modality that appeared nowhere in any NCCN recommendation for that scenario.

The problem is that the study tested GPT-3.5, and it was submitted for publication in early 2023.

Three years is a long time. Citing that number today as evidence about current models is closer to arguing against next-generation sequencing using a 2015 panel — the technology being criticised no longer exists in the form that was tested. And clinicians notice. Every time a talk opens with that slide, a portion of the room quietly concludes that the presenter has not used these tools recently, and stops listening. Which is unfortunate, because the reliability problem is real. It has simply moved.

What current models actually do

Better than most sceptics expect. Considerably better.

A recent analysis put ChatGPT-4o against the NCCN Rectal Cancer Guidelines, version 4.2024. The design was reasonable: 135 clinical questions constructed from 45 pages of guideline text, each response independently rated by two physicians on a five-point scale, with disagreements adjudicated by a board-certified surgical oncologist. The model showed high concordance across every domain assessed, and the authors noted marked improvement over prior ChatGPT iterations their own group had tested.

That pattern repeats across tumour sites. Comparative work in gastrointestinal oncology found GPT-4 and Gemini performing credibly against NCCN across six tumour types (Bresler et al, J Surg Oncol 2025;132(2):317–322). In advanced gastric cancer, GPT-4o outperformed Gemini Advanced in supporting multidisciplinary decision-making (Li et al, Eur J Surg Oncol 2025;51(8):110096). A retrospective breast cancer analysis found GPT-4o reached full concordance with tumour board decisions in 31 of 33 cases when given structured clinical input, with the two partial mismatches turning on genetic counselling rather than treatment selection.

If your mental model is still "it confidently invents cancer treatments," that model is out of date.

The number nobody puts on the slide

Here is what almost never gets shown alongside the AI concordance data.

Real-world clinician concordance with NCCN guidelines runs somewhere between 65% and 80%. For certain lymphoma subtypes it falls to between 11% and 46%. Eighty-eight percent of clinicians report using NCCN guidelines for Hodgkin lymphoma, yet adherence to PET-adapted approaches remains inconsistent (Yost et al, Blood 2025;146(Suppl 1):7914). Rectal cancer adherence sits around 60–70%.

Guideline concordance is genuinely difficult. It is difficult for us, working from the same guidelines, with training and colleagues and time to think.

This matters for how the question gets framed. "Is the model perfect?" is the wrong test, because nothing clears it. The useful question is whether a given tool makes guideline-concordant care more likely than what happens now — a busy clinic, a half-remembered version of a guideline that was updated four months ago, and no time to check.

That is a more honest starting point, and also a more demanding one. It means the burden does not sit on AI to be flawless. It sits on any tool claiming to help to demonstrate that it actually does.

So where does it still break?

Three places. None of them is raw medical knowledge, and that is the thing worth understanding.

The errors hide in the branches

The rectal cancer analysis found something more useful than an accuracy percentage. The errors were not distributed evenly across the question set. They clustered in complex conditional treatment algorithms — the branching pathways covering neoadjuvant sequencing, total neoadjuvant therapy, organ preservation decisions, surveillance intervals after a complete clinical response.

The authors interpret this as these models being strong at discrete factual recall while struggling with multi-step reasoning under clinical uncertainty.

Sit with that, because it is exactly inverted from what you would want. Discrete factual recall is the part you can settle in thirty seconds with a search. The multi-step conditional reasoning is the part you actually opened the tool for — the third branch down, where the patient has one comorbidity that changes the calculus and the guideline hands you a decision node rather than an answer.

It also means headline accuracy figures systematically mislead. Any realistic question set contains far more simple questions than hard ones, so the mean is pulled upward by exactly the queries where you needed no help. A model reporting 90% accuracy might be near-perfect on the easy 80% of questions and considerably worse on the 20% that brought you there.

It agrees with you

This is the finding that should change how anyone uses these tools, and it has not improved with model scale.

Omar and colleagues at Mount Sinai built 300 physician-validated clinical vignettes. Each contained exactly one fabricated element — an invented laboratory test, a physical or radiological sign that does not exist, or a made-up medical condition. Everything else was clinically sound. They ran these past six leading language models and counted how often each model repeated or elaborated on the planted fiction.

Up to 83% of the time, depending on the model.

They then tried the obvious mitigation: a prompt explicitly warning the model that the vignette might contain false information. That roughly halved the rate. It did not eliminate it (Commun Med 2025;5(1):330).

Consider how these tools are used in practice. You are between patients. You half-remember that there is a threshold for something — an eGFR cutoff, a CPS score, a nodal count — and you type the question with your recollection embedded in it. "In a patient with CPS 12, is pembrolizumab..." The model does not challenge your premise. It builds on it, fluently, in the same confident register it uses when it is correct.

The tool amplifies precisely the error you consulted it to correct. Nothing in the output signals that this has happened, because the output looks identical either way.

The practical implication is a habit change rather than a tooling change: stop putting your recollections into the question. Supply the clinical picture and let the tool retrieve the threshold. If you supply the number yourself, you have contaminated the answer before it began.

The citations still do not hold

This is the failure that has not tracked capability at all, and it is the strangest of the three.

Chelli and colleagues tested reference generation against 11 published systematic reviews across four fields, analysing 471 generated references. A reference counted as hallucinated if any two of title, first author or year were wrong. Hallucination rates came in at 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard. Precision — the proportion of generated references that were both real and relevant — was 9.4%, 13.4% and 0% respectively (J Med Internet Res 2024;26:e53164).

Two details from that paper matter more than the headline rates. First, the fabricated references read as entirely genuine: real authors who publish in the relevant field, real journals covering that topic, plausible titles in the right era. Second, the authors documented geographical and open-access bias in the papers the models did retrieve correctly — a point I will come back to.

The mechanism explains why plausibility checks fail. A citation has an extremely regular shape — surname, initials, title, journal, year, volume, pages. A next-token predictor can generate a well-formed instance of that shape with no corresponding object existing anywhere. The output is not a lookup that went wrong. It is a successful act of synthesis applied to entirely the wrong task.

The only test that works is resolution. Search the DOI. Search the exact title in PubMed. Nothing about how a reference reads will tell you.

What the fix looks like, and the evidence for it

If these failures were about medical knowledge, the answer would be larger models. They are not, so it is not.

Two experiments from the same group tested the intervention you would expect. Rather than asking a model to recall guidelines, they indexed the guidelines and required the model to retrieve before answering.

The lymphoma version indexed the 2025 NCCN Lymphoma Guidelines into a vector database and ran 50 clinical vignettes through a retrieval-augmented GPT-4 agent, comparing against baseline GPT-4. Retrieval significantly improved treatment accuracy and reduced deviation from evidence-based care (Blood 2025;146(Suppl 1):7914). The breast cancer version, indexing the 2025 NCCN Breast Cancer Guidelines, found the same — better concordance, fewer hallucinations (Clin Cancer Res 2026;32(4 Suppl):PS3-06-29).

Both are conference abstracts with small samples, and the authors state plainly that larger validation is needed. Do not over-read them. But the direction matches the mechanism, which is the reason to take them seriously: if the problem is that the model generates from compressed memory rather than fetching a document, then requiring it to fetch a document should help. It does.

Where OneRX differs

OneRX is a medical AI platform for clinical decision support, and it is built directly against the three failure modes above rather than around them. Each design decision maps to a specific documented problem.

In building retrieval for Indian oncology practice, the failure we encounter most often is not wrong medicine. It is correct medicine for a drug that is not on the shelf, not on the formulary, or not within reach of the patient sitting in front of you. That is the gap a clinical decision support system built for this market has to close, and it is not a translation problem.

There is also a behavioural tell worth knowing, and it applies to any AI clinical reference you evaluate, including this one. A retrieval system can return nothing, because sometimes the corpus genuinely does not contain an answer. A model working from memory always has something to say. If a tool has never once told you it could not find anything, it is not searching.

The part no benchmark measures

Every study cited above scores against NCCN. Which is reasonable if you practise in Pennsylvania.

For a clinician in Pune, NCCN concordance is perhaps half the answer. The rest of it: is the agent approved by CDSCO, and does the Indian label cover this indication? Is it procurable this month, or is the distributor out? What is the NPPA ceiling price under DPCO, and does a generic or biosimilar change the arithmetic entirely? Does the regimen map to an AB-PMJAY health benefit package, and will preauthorisation clear? And what does the National Cancer Grid recommend at this resource level — essential, optimal, or optional — given that its tiers are built to link to the PMJAY package structure for preauthorisation and claims?

Recall the finding from Chelli's group about geographical and open-access bias in what these models retrieve. That bias is the mechanism here. Indian regulatory, pricing and coverage material barely exists in the English-language corpus these models were trained on, so the model does not recall it, does not flag its absence, and returns a US answer with full confidence.

The National Cancer Grid produces India-specific consensus guidance rather than adopting international guidance wholesale, precisely because implementing standard-of-care treatment in a low- and middle-income setting raises challenges the source guidelines were not written to address (Chopra et al, J Glob Oncol 2018;4:1–15). An engine indexing NCCN alone will not surface that reasoning, because it sits outside the source being indexed.

A recommendation you cannot deliver is not a partially useful answer. It is a wrong answer that reads like a correct one. Closing that gap — global evidence with Indian applicability attached — is what OneRX was built for, and it is why 1,400+ doctors across 8+ countries use it at the point of care.

What to do on Monday

Five things, none of which require abandoning these tools.

1. Open one citation at random, rather than all of them. Fabrications rarely travel alone, and a single spot check tells you what kind of system you are dealing with.
2. Ask which guideline version the answer reflects. If it cannot name one, it is answering from memory, and you have no way to know how old that memory is.
3. Do not put your recollection into the question. This is the highest-yield habit change available, and the 83% figure is why.
4. Push into the conditional branches deliberately. The performance you care about is not at the trunk.
5. Ask something jurisdictional — the CDSCO status of a recently approved agent. Silence, or a US answer delivered as though it were universal, tells you the tool was built for somewhere else.

Questions clinicians actually ask

Is ChatGPT reliable for medical advice in 2026?

More reliable than the widely quoted 2023 assessments suggest — GPT-4o showed high concordance with NCCN rectal cancer guidelines across 135 physician-rated questions. It remains unreliable for complex conditional pathways, elaborates on false details embedded in your question in up to 83% of tested cases (Commun Med 2025;5(1):330), and generates citations that frequently do not resolve.

Are the older studies useless now?

Their error rates are obsolete. Their description of error shape is not. Chen et al found non-concordant recommendations sitting inside otherwise correct responses (JAMA Oncol 2023;9(10):1459–1462), which is why checking two lines out of five fails regardless of which model produced them.

Does retrieval-augmentation genuinely help, or is that marketing?

There is published evidence, though it is early. Indexing NCCN guidelines and retrieving before answering improved concordance and reduced hallucination in both lymphoma and breast cancer experiments (Blood 2025; Clin Cancer Res 2026). Both are small conference abstracts pending full publication and should be read as such.

How do I check a citation quickly?

Copy the exact title into PubMed, or resolve the DOI. Do not assess it by reading it — fabricated references use real authors from the right field in real journals, so they read perfectly.

Why do these tools miss Indian context so completely?

Training corpora over-represent US and Western European, open-access literature — a bias documented directly in reference-generation testing (J Med Internet Res 2024;26:e53164). CDSCO status, NPPA pricing, PMJAY packages and NCG resource-stratified tiers are thinly represented, so there is little for a model to recall, and no mechanism prompts it to flag that its answer is jurisdiction-specific.