~/shanegraffiti.com/research/trust-but-verify
Shane Graffiti Inc. AI Research Division 2026
Five open-access LLMs, asked 103 clinical multiple-choice questions where the historically correct answer is now a banned or withdrawn drug, hallucinated dangerous recommendations 90–99% of the time. A five-agent adversarial auditing loop one model, five rotating personas, a real-time regulatory web check, and up to three retry attempts cut that hallucination rate by roughly 53% across every model tested, by trading raw accuracy for safe refusal. Even proprietary frontier models with native browsing kept recommending withdrawn drugs anyway, because retrieval without an adversarial check just hands the model evidence it's free to ignore.
A drug that was the textbook-correct answer to a clinical question can be banned the following year, but the LLM's parametric knowledge doesn't know that it keeps matching the question to whatever pattern its training data rewarded. This isn't a hypothetical edge case: a 2025 case of bromism was traced directly to a patient following ChatGPT-generated instructions, and a kidney-transplant patient in Hyderabad lost the transplanted organ after stopping antibiotics on the strength of a misleading AI response. Meanwhile adoption keeps climbing over 40 million people a day ask ChatGPT health questions, and two-thirds of US physicians used AI tools clinically in 2024, up from 38% the year before.
One LLM backbone, five rotating personas achieved through prompt redirection rather than five separate models. A query enters through the Router and either skips straight to a lightweight chat agent or gets pulled into the adversarial auditing loop clinical reasoning, structured extraction, and a live regulatory check with up to three retries before the system is willing to issue a final answer.
{{ s.desc }}
Every model, asked directly, identified the textbook-correct option with high accuracy and that's exactly the problem, because the textbook-correct option is the one that's been banned. Hallucination Error Rate clustered at 90–99% across the board in vanilla mode. Routing the same five models through the adversarial loop dropped accuracy hard (the system is now refusing instead of confidently answering) while cutting HER by roughly half and pulling the Pointwise Score back toward zero from its −0.25 floor.
Accuracy dropping isn't a failure here it's the intended trade. Every percentage point lost is a dangerous recommendation converted into an appropriate refusal, and Component Fidelity staying between 73.82% and 85.71% across all five models confirms the router and auditor agents are reliably catching what they're supposed to catch.
Proprietary models with native browsing and reasoning weren't immune in two recorded interactions, the model's own search results surfaced the withdrawal, and it recommended the drug anyway.
Search results explicitly surfaced the drug's withdrawn status. The model acknowledged it in its reasoning trace, then selected it anyway as the final clinical recommendation retrieval without an adversarial check just handed the model evidence it was free to ignore.
None of this required a bigger model or a fine-tuning run. The entire safety gain comes from architecture how the same backbone is decomposed and re-queried not from new parameters.