Evidence Brief · OneRx

Evidence Synthesis AI: Six Tests Before You Trust It

Published 26 July 2026 · Reading time 10 min
By the OneRX Clinical Content Team · Reviewed against source publications prior to release
For healthcare professionals only

On this page: Why the phrase stopped meaning anything · Six properties · The ten-minute test · Where OneRX differs · FAQ · References

Sit through three vendor demonstrations in a week and you will hear the same phrase in all three. Evidence-based. Grounded in the literature. Built on peer-reviewed sources.

Why the phrase stopped meaning anything

None of it means anything as stated, because a model trained on medical literature can say all three truthfully while still generating recommendations from compressed memory. A physician who read the literature once and has practised from recollection ever since is not doing evidence-based medicine either, and for the same reason.

What follows is an attempt to make the phrase testable. Six properties, each of which a platform either has or does not, and each of which you can check yourself in a few minutes. The sixth is specific to Indian practice and is the one most platforms have never considered.

Six properties

Property one — retrieval rather than recall

The architectural question that determines everything downstream: when the system answers, is it fetching a document or generating from parameters?

A model answering from parametric memory has compressed a large body of literature into weights. It cannot tell you which paper informed a given sentence, because there is no discrete "which" to point at. A retrieval-grounded system queries an indexed corpus at the moment you ask, retrieves specific documents, and generates conditioned on their contents.

This is not a theoretical preference and there is now evidence behind it. Indexing the 2025 NCCN Lymphoma Guidelines into a vector database and running 50 clinical vignettes through a retrieval-augmented GPT-4 agent significantly improved treatment accuracy and reduced deviation from evidence-based care compared with baseline GPT-4 (Yost et al, Blood 2025;146(Suppl 1):7914). The parallel breast cancer experiment, indexing the 2025 NCCN Breast Cancer Guidelines, reported the same direction of effect (Yost et al, Clin Cancer Res 2026;32(4 Suppl):PS3-06-29).

Both are conference abstracts with small samples and the authors say so plainly. Do not over-read them. But the direction matches the mechanism, which is what makes them worth citing at all.

There is a behavioural tell, and it is the fastest diagnostic available. A retrieval system can return nothing, because sometimes the corpus genuinely does not contain an answer to what you asked. A model working from memory always has something to say. If a platform has never once told you it could not find anything, it is not searching.

Property two — citations that resolve

Not citations that look right. Citations you can open.

Reference-generation testing across 11 published systematic reviews and 471 generated citations found hallucination rates of 39.6% for GPT-3.5, 28.6% for GPT-4 and 91.4% for Bard, with a reference counting as hallucinated if any two of title, first author or year were wrong. Precision — references that were both real and relevant — came in at 9.4%, 13.4% and 0% respectively (Chelli et al, J Med Internet Res 2024;26:e53164).

The detail that matters more than the rates is how the fabrications read. Real authors who publish in the relevant field. Real journals covering that topic. Plausible titles from the right era, correctly formatted. Nothing on the surface of a fabricated citation distinguishes it from a real one, which is precisely why reading it more carefully does not help.

The test takes ninety seconds and almost nobody runs it. Open three citations. If one fails to resolve, misattributes, or lands on a paper that does not support the specific claim it was attached to, the platform is generating bibliographies rather than retrieving them.

Property three — a declared source hierarchy

Evidence conflicts constantly, and a system that hides how it resolves those conflicts is asking for trust it has not earned.

Consider a routine situation. A phase III RCT reports a result. A meta-analysis including it reaches a slightly different conclusion. A guideline has not yet incorporated either. The regulatory label predates all three. And a national consensus document takes a different position again for resource reasons. All five are simultaneously current, and they disagree.

An evidence synthesis AI should declare how it ranks these and apply that ranking where you can see it. Ours, stated plainly:

1. Regulatory label for the relevant jurisdiction, establishing what is permissible
2. Current guideline version from the specialty-appropriate body
3. Phase III RCT and meta-analytic evidence
4. Phase II and single-arm data, flagged as such
5. Consensus and expert opinion, flagged as such

You do not have to agree with that ordering. You do have to be able to see it. A platform that will not state its hierarchy is running an undisclosed editorial policy on your behalf.

Property four — recency with version pinning

"Latest evidence" is not a claim. It is a date, and a system that cannot produce the date is not making a claim at all.

NCCN Guidelines are revised multiple times per tumour site per year. Between a plenary presentation in June and a guideline revision, the correct answer to a first-line question can change inside a single quarter.

Three things a platform should be able to state without being prompted: which guideline version the answer reflects, when the corpus was last indexed, and whether anything published since that date remains unincorporated. A model answering from training weights can supply none of the three, and — this is the part that matters clinically — it will state superseded guidance in exactly the same confident register it uses for current guidance.

Property five — divergence surfaced rather than flattened

This is where most platforms quietly fail, and they fail because flattening produces a cleaner-looking answer that demonstrates better.

Guideline bodies genuinely disagree. On thresholds, on sequencing, on whether an intervention justifies its toxicity in a particular population. Collapsing that disagreement into one confident recommendation destroys exactly the information a specialist opened the tool to find.

The National Cancer Grid is a useful case. It produces India-specific consensus guidance rather than adopting international recommendations wholesale, because implementing standard-of-care treatment in a low- and middle-income setting raises challenges the source guidelines were not written to address (Chopra et al, J Glob Oncol 2018;4:1–15). That is a considered, published position from the body guiding care across a large share of India's cancer centres. Averaging it away with NCCN into a single answer is not synthesis. It is data loss.

Property six — the deliverability layer

For Indian practice there is a sixth requirement that no Western-built platform carries, and it is not an enhancement. A guideline-concordant recommendation that cannot be delivered is not partially useful. It is wrong in a way that reads as right.

The NCG's essential, optimal and optional stratification is explicitly linked to the AB-PMJAY health benefit package structure for preauthorisation and claims documentation (PMC9396157). That is a parallel evidence structure built for real constraints, and it has to be indexed as a primary source rather than appended as regional colour.

Where OneRX differs

OneRX is a medical AI platform for clinical decision support built against these six properties rather than around them.

The first five are table stakes for any serious clinical decision support software and we do not present them as a differentiator. The sixth is what makes an answer actionable in Kochi or Kanpur rather than only in Boston, and it is why 1,400+ doctors across 8+ countries use OneRX at the point of care.

The ten-minute test

Run these on whatever you currently use, including this platform.

1. Ask something with genuinely thin evidence — a rare sequencing question where the literature does not support a confident answer. Does the system say so, or does it produce one anyway?
2. Open three citations at random and resolve them in PubMed.
3. Ask which guideline version the answer reflects, and when the corpus was last indexed.
4. Ask about a question where guideline bodies are known to disagree. Divergence, or a single flattened recommendation?
5. Ask a regulatory question — the CDSCO status of a recently approved agent. Silence, or an FDA answer delivered as universal, tells you where the tool was built.
6. Re-ask the first question in materially different words. Substantial divergence between phrasings indicates recall rather than retrieval.

Clear all six and you are looking at evidence synthesis. Clear three and you are looking at fluent medical writing, which is a genuinely useful product and a completely different one. Price and trust it accordingly.

Questions clinicians actually ask

What makes an AI evidence-based rather than just trained on medical text?

Retrieval architecture. Training compresses literature into weights; an evidence synthesis AI queries an indexed corpus at answer time and can name the documents it used. The practical marker is whether the system is capable of returning an empty result.

Does retrieval-augmentation measurably improve accuracy?

Published evidence points that way. Indexing NCCN guidelines and retrieving before answering improved concordance and reduced hallucination in both lymphoma and breast cancer experiments (Blood 2025; Clin Cancer Res 2026). Both are small conference abstracts pending full publication.

How do I check whether citations are real?

Resolve them. Search the exact title or the DOI in PubMed. Reading them more carefully does not work, because fabricated references use real authors from the right field in real journals.

Should an AI tell me when guidelines disagree?

Yes, and the absence of disagreement on a contested question is itself a warning sign. NCCN, ESMO and the National Cancer Grid take genuinely different positions on some questions, each with published reasoning.

What is the NCG essential, optimal and optional classification?

The National Cancer Grid's resource-stratified recommendation tiers, designed for Indian implementation and linked to AB-PMJAY health benefit packages for preauthorisation and claims.