How a $12M HealthTech Startup Nearly Triggered a Hospital Incident Using an LLM

How a real-time discharge assistant almost published the wrong medication dose

In Q1 2025, a HealthTech startup with $12 million in Series A funding and 42 engineers launched a pilot of a real-time patient discharge assistant. The product ingested physician notes and returned plain-language discharge instructions, medication lists, and follow-up schedules. The commercial stakes were high: the product was integrated into three partner hospitals and was slated to roll out to 18 more facilities in 90 days.

The deployment relied on a third-party large language model (LLM) API. The vendor whitepaper claimed “near-zero hallucinations” on internal tests, with a one-page appendix listing a handful of unannotated sample prompts. Engineering leaders felt pressure to move fast: sales had promised an April launch, operations had scheduled one week of clinician training, and investors expected early traction metrics by the next board meeting.

Within two weeks of the pilot, an on-call nurse discovered an instruction that recommended a twice-daily dose of a medication class for which the patient’s chart showed an allergy. The error did not cause harm, but the near-miss triggered a full product halt while the incident was investigated. For CTOs and engineering leads evaluating LLMs for safety-critical deployments, this case exposes how vendor claims can look reassuring but mask methodological gaps. The rest of this case study breaks down what happened, how the team changed course, and what measurable results followed.

The clinical hallucination problem: why standard vendor tests failed to prepare us

The startup suprmind.ai relied on a vendor whitepaper claiming “<0.1% hallucination" on an undisclosed benchmark. Senior engineers took that as a green light. The audit after the incident uncovered three linked problems:

• Definition mismatch: The vendor defined “hallucination” narrowly as fully fabricated facts about common public knowledge, not as clinically dangerous misstatements like incorrect dosages, missing allergies, or erroneous prescription durations.
• Non-adversarial sampling: The vendor tests used templated prompts and public QA datasets. They excluded adversarial or ambiguous clinician phrasing typical in real notes.
• Single rater adjudication: The vendor used an internal reviewer to mark outputs as “correct” or “incorrect” without cross-checking against clinical ground truth or external raters, inflating apparent accuracy.

We found additional issues that typically get missed when teams rely on vendor metrics:

Data leakage: Some test questions were discovered to be in the vendor model’s training data, which artificially reduces hallucination rates.

Prompt sensitivity: With small prompt edits the model began to hallucinate at much higher rates. The vendor used a single prompt template and did not report sensitivity analysis.

Sampling parameters: The vendor evaluated with deterministic decoding (temperature 0) but the production system had varied settings to improve readability, increasing hallucination risk.

Concrete numbers from the vendor and why they conflicted with our field results

The vendor reported 0.08% hallucination on their benchmark. In our first week of pilot traffic, we observed clinically actionable misstatements at a rate of roughly 0.98% of discharged patients – a 12x difference. The discrepancy came from inconsistent definitions, sample selection, and production differences in prompts and decoding.

A safer model pipeline: combining retrieval, verification, and conservative models

After the incident the engineering and ML teams paused the rollout and agreed on a new risk-first strategy. The guiding principle was simple: stop trusting a single opaque score from a vendor. Instead, build a multi-layer pipeline that treats the model as one component in a chain that must be auditable.

The chosen stack had four elements:

• Deterministic grounding – require asserted clinical facts to cite a vetted source from the hospital electronic health record (EHR) or a local medication database before being presented.
• Model voting with diversity – run two different model families (a closed API model and an open-weight local model) and accept an assertion only if both agree and align with the retrieved evidence.
• Rule-based filters for critical tokens – block outputs that change dosage units, routes, or allergy status without explicit, verifiable data points.
• Human-in-the-loop verification for edge cases – route any output flagged by the voting or filters to a nurse reviewer before release.

Why these choices

Retrieval reduces reliance on the model to remember facts. Voting exposes model divergence that often precedes hallucinations. Rules prevent catastrophic changes of critical attributes that downstream systems rely on. Human review provides a final safety net for low-frequency but high-impact cases.

Deploying the safety stack: a 120-day engineering timeline

The team adopted a 120-day plan with weekly milestones and measurable gate criteria. The timeline split into four 30-day sprints.

Day 0-30: Audit, instrumentation, and baseline testing

• Performed a 10,000-query retrospective audit across pilot notes (test window: March 10-20, 2025).
• Defined a strict adjudication rubric: clinically actionable misstatement = any output that, if followed, could change treatment or increase patient risk.
• Instrumented logging to capture prompts, model outputs, metadata (model name, temperature, token usage), and downstream actions.

Day 31-60: Grounding layer and rule engine

• Built a retrieval module that fetched EHR fields and the hospital medication formulary for each patient. Retrieval results were cached for 48 hours.
• Implemented rule engine to reject outputs that modified dosage magnitude by more than 10% without a matching EHR entry.

Day 61-90: Dual-model pipeline and voting

• Integrated API model A (vendor closed model; evaluation snapshot: GPT-4, tested March 15, 2025) with a local LLM B (Llama 2 70B finetuned on clinical notes, snapshot Feb 2025).
• Defined voting logic: accept statement only if both models agree on the normalized fact and the retrieved grounding supports it.

Day 91-120: Human review workflow and monitoring

• Created a clinician dashboard that displayed the retrieval evidence, both model outputs, and the voting result. Built a single-click approve/reject flow.
• Set monitoring alerts: any rejected output triggered an incident and a root cause review within 24 hours.

From nine monthly critical incidents to 0.2: measurable results in six months

After 180 days of the new stack running in production (pilot extended), the team measured the impact with the predefined clinical-adjudication rubric. Results were logged from April 1 to September 30, 2025.

Metric Pre-change (pilot) Post-change (6 months) Monthly clinically actionable misstatements 9 0.2 Fraction of discharges requiring human review N/A (no review) 3.5% Average latency (user-facing) 120 ms 360 ms Cost per request (inference + retrieval) $0.12 $0.48 False-positive rate (safe outputs flagged) N/A 1.0%

Key takeaways from the numbers:

• The clinically actionable misstatement rate fell from 9 per month to 0.2, a 45x reduction. This is measured against the startup’s strict rubric; less strict definitions would show smaller improvements.
• Human review coverage landed at 3.5% of discharges. That limited headcount needs while keeping high-sensitivity checks on rare edge cases.
• Costs increased approximately 4x per request, and latency increased by about 240 ms on average. The product team accepted these trade-offs because the alternative was unsafe.

Why conflicting vendor numbers still persist

Despite our regression tests being public to partner hospitals, vendors continued to publish low hallucination rates. The conflict exists because of different evaluation axes: the vendor measured a narrow class of fabrication, while our metric focused on clinical actionability. Both can be “true” but are not comparable. For decision makers, the lesson is to map vendor metrics to your domain-specific harms rather than accept headline numbers.

5 hard lessons engineering leaders must internalize before productionizing LLMs

Demand the evaluation rubric, not a single number. Know precisely what the vendor calls a hallucination.

Test on real production-style prompts with adversarial edits. Models are brittle when prompts vary in small ways.

Run multi-model disagreement tests. Divergence between models often signals uncertainty even when both look plausible.

Instrument everything and keep examples. Recreate incidents with the exact prompt, model version, and decoding parameters.

Budget for operational controls. Human review, retrieval indices, and logging cost money and add latency; plan for those trade-offs early.

Thought experiments to sharpen decision making

Try these mental exercises with your team:

• Imagine a model output that changes an antibiotic dose by 25% without a matching EHR entry. How many such outputs per 10,000 discharges would you tolerate before halting the pipeline? Map that tolerance to required detection sensitivity.
• Suppose the vendor gives you a “0.01% hallucination” claim. Ask them to produce a stratified sample: show 1,000 raw prompts they used and the exact model parameters. If they refuse, treat the claim as unsupported.
• For cost trade-offs: if your RAG+voting approach increases per-request cost by 4x, calculate the budget to scale to 100,000 monthly requests. Decide if you will charge, absorb costs, or tighten review thresholds.

How your engineering team can reproduce our tests and safety checks

If you are a CTO, engineering lead, or ML engineer evaluating models for safety-critical contexts, here is a reproducible checklist and minimal experiments to run. Each item maps to concrete decisions and numbers.

Checklist and experiments

Assemble a domain-specific adversarial test set: collect at least 5,000 representative prompts from real logs, then add 1,000 adversarial perturbations (typos, truncated notes, ambiguous phrasing).

Define the harm rubric with clinical stakeholders. Quantify “clinically actionable” and document examples for adjudicators.

Run each candidate model with identical prompts and parameters. Log model name, version, temperature, top-p, token limits, and full outputs. Test dates: run initial tests over a 10-day window and record the snapshot timestamp.

Use at least two independent human raters per output. Report inter-rater agreement. If agreement is low, expand the rubric and adjudicate disagreements with a clinician panel.

Measure model sensitivity to prompt variations: for 500 prompts, run a 10x perturbation matrix and record hallucination variance.

Calculate production metrics: expected human review rate, additional latency, and cost per request under your traffic assumptions.

Design gating rules: set conservative thresholds for automatic release versus human review, and test false-positive/false-negative trade-offs.

Example minimal report format to demand from vendors

Field Required contents Model snapshot Model name, version, training cutoff date, and weights hash where applicable Evaluation rubric Precise definitions of hallucination classes and adjudication rules Test set Size, provenance, and whether any items overlap training data Decoding params Temperature, top-p, beam size used for evaluation Raw outputs Samples of prompts and unredacted outputs (or redacted but reproducible) with labels

Demanding transparency forces vendors to align their claims with your use case. The market will always produce headline numbers. Your job is to translate those numbers into domain-specific risk budgets and engineering plans.

Final pragmatic checklist before production

• Have a reproducible test harness and run it on every model update.
• Lock prompts and decoding parameters in production; treat any change as a new release with regression tests.
• Implement affirmative grounding – never present asserted clinical facts without verifiable evidence links.
• Plan for human review capacity scaled to the expected edge-case rate, not to average traffic.
• Set up rapid rollback procedures tied to monitored incident thresholds.

In domains where hallucinations have real-world consequences, vendor claims are a starting point, not an endpoint. The $12M startup in this case study avoided a serious patient safety incident only after adopting multiple layers of verification and conservative operational choices. The result was a measurable drop in critical errors, at the cost of higher latency and expense. For responsible deployment, those trade-offs are the point: safety systems are what prevent the single hallucination that can devastate reputation, invite regulatory scrutiny, or harm patients.