When a Healthcare Startup’s Virtual Assistant Leaked Patient Notes: Ravi’s Night Shift

When Ravi’s Triage Bot Answered the Wrong Question

Ravi was on call at 02:00 when a support engineer woke him. The company’s patient triage assistant – a lightweight conversational agent – had responded to a seemingly harmless user prompt with chunks of internal patient notes. The user was not an authorised clinician. No keys had been stolen. At first glance it looked like an unlucky bug: a prompt somehow pulled the wrong context. Meanwhile the incident triggered a cascade of meetings, log reviews and emergency patches.

As it turned out, the assistant had been coaxed into exposing the notes through a crafted prompt that overrode safety instructions embedded in the system prompt. The engineering team had tested for prompt injection early in development and assumed they were covered. This led to https://londonlovesbusiness.com/the-10-best-ai-red-teaming-tools-of-2026/ a painful realisation – the test had been narrow. It focused on single-turn injection attempts and ignored multi-turn conversations, chained prompts and subtle instruction confusion. Ravi’s night shift became a turning point for the organisation’s security approach.

The Hidden Risk of Treating Prompt Injection as a One-Off Test

Prompt injection is often framed as a single class of attack: an attacker embeds instructions in input that cause a model to ignore higher-level safety constraints. That framing is useful, but incomplete. In practice, the attack surface is wider. The model interacts with retrieval systems, business logic, external tools and human operators. An exploit might begin with a small, innocuous prompt and evolve into a multi-step sequence that bypasses safeguards.

Organisations that test only for obvious injection examples tend to feel safe. They run a few unit tests, check that “do not reveal private data” works, then move on. That creates a false sense of security. A few things are easy to miss:

• Stateful interactions. Multi-turn conversations can gradually manipulate the model’s behaviour.
• Retrieval augmentation. The model may combine retrieved documents with user input in unexpected ways.
• Tool use. If the model can call functions or external APIs, attackers can craft prompts that trigger data exfiltration via legitimate channels.
• Model updates. Behaviour changes with model upgrades and new fine-tuning, so yesterday’s tests may not hold tomorrow.

These blind spots mean that prompt injection is not a single test you run once. It is part of a broader threat model. Treating it as otherwise invites incidents like the one Ravi faced.

Why Standard Prompt Tests Often Miss the Real Threats

Engineering teams usually approach model safety with a checklist. They add a system prompt that instructs the assistant to refuse sensitive requests. They implement an input filter for certain keywords. They run a handful of handcrafted attack prompts. That is a sensible start, but the approach is brittle.

Here are common failure modes we see in practice:

• Overfitting to test cases – defences that work against the team’s test prompts but fail against slightly different wording.
• Silent degradation – models may produce harmless but subtly biased or revealing content when under ambiguous prompts.
• Tool chaining blind spot – when the model can call functions, attackers can route data through the function outputs rather than the model response.
• Operational drift – configuration changes, new features and model upgrades change behaviour without corresponding re-tests.

As a security researcher, I admit to making these mistakes. Early on, my red team ran impressive injection examples that earned a few quick wins. Then a real user created an attack that looked nothing like our tests. We had validated the wrong things. That humbling experience pushed us to rethink red teaming not as optional penetration testing but as an essential, ongoing practice.

Foundational understanding: what red teaming brings that unit tests do not

Red teaming treats models as adversarial systems. Rather than checking isolated rules, it simulates how an intelligent opponent might probe and exploit the entire application stack. That includes:

• Threat modelling: identifying assets, attacker goals and possible pathways to compromise.
• Adversarial prompt generation: using creative, human and automated techniques to find weak spots.
• Tool and data flow analysis: observing how data moves between retrieval systems, models and APIs.
• Continuous assessment: integrating tests into CI pipelines so findings surface as the system changes.

How Red Teams Found the Real Weakness and Changed the Process

After the incident Ravi faced, the company set up a small red team. They used a mixture of manual testers and free, open-source tools. Initially, stakeholders wanted enterprise platforms – those promise comprehensive dashboards, compliance reports and managed testing. The red team proposed a pragmatic, cheaper path: prove the concept with inexpensive tools, then scale if needed.

The red team ran three types of exercises:

Human adversarial sessions. Skilled testers played the role of attackers and attempted both direct and indirect manipulations.

Automated mutation tests. Scripts generated thousands of prompt variants by paraphrasing, adding noise and chaining instructions.

Integration tests. They simulated tool calls, retrieval responses and partial disclosure scenarios to see how the system reacted in end-to-end flows.

As it turned out, the integration tests were the most revealing. A combination of retrieval snippets and carefully timed follow-up prompts allowed the testers to reconstruct the internal context. The model did what models do well – combine bits of information – and this behaviour was turned against the system.

This led to a series of practical changes:

• Context sanitisation: any retrieved text that might contain sensitive data was redacted or replaced with metadata before being fed to the model.
• Phased tool access: functions with sensitive outputs required a secondary authorisation token, not just a model call.
• Continuous labelling: logs were enriched with tags for each conversation turn so unusual patterns could be detected automatically.

Why free tools can be as effective as enterprise platforms

There is a widespread belief that you need an expensive platform to run meaningful red team exercises. That is not universally true. Free tools and open-source models can accomplish a lot if you understand what they are good for and their limits.

Free tool advantages:

• Fast iteration – you can build quick harnesses and iterate on attack ideas without procurement delays.
• Transparency – open-source models let you inspect behaviour more closely and reproduce failure modes.
• Cost efficiency – for many organisations a proof of concept is enough to inform design and justify later investment.

Limitations to acknowledge:

• Scale and management features – enterprise products provide monitoring, compliance logs and support that reduce operational overhead.
• Responsibility and indemnity – platforms sometimes include contractual protections and certified processes that certain sectors require.
• Integration complexity – an enterprise solution often arrives with connectors and workflows that save time on complex systems.

Characteristic Free/Open Tools Enterprise Platforms Cost Low; pay for infrastructure Higher; subscription model Customisability High Medium to high Scale & Support Depends on in-house expertise Vendor support and SLAs Compliance Features Limited; requires internal controls Often stronger, built-in

From a Panic Fix to Reliable Defences: Real Results After Red Teaming

Six months after Ravi’s incident, the company could point to measurable outcomes. Not all were glamorous, but they mattered: fewer incidents, faster incident response and clearer risk metrics for product managers.

Concrete results included:

• Reduction in successful simulated exfiltration attempts by more than 80% in controlled tests.
• Deployment of automated regression tests that ran on each model update, catching regressions within minutes rather than weeks.
• Improved cross-team collaboration – product, engineering and legal now shared threat models and response plans.

Most important was a cultural shift. Security stopped being a gate that blocked features and became part of product conversation; the team accepted that models change over time and testing must be continuous. This changed priorities in planning cycles and budget discussions. When the company later evaluated enterprise offerings, they did so with concrete data about which capabilities mattered most to them.

Practical blueprint: how to start red teaming with limited budget

If you are responsible for a model-driven product and you cannot afford an enterprise red team subscription, here is a practical blueprint used by teams that succeeded.

Build a minimal threat model. Identify sensitive assets and likely attacker goals.

Assemble a small team of testers – two or three people familiar with prompt design and software integration.

Create an injection harness. Use open-source language models and scripts to automate prompt variations and multi-turn sequences.

Prioritise end-to-end scenarios. Test retrieval, tool calls and system prompt interactions, not just single-turn examples.

Automate regression checks. Add failing cases to CI so they run on relevant changes.

Log liberally. Capture conversation turns, returned tokens and retrieval results for post-mortem analysis.

Escalate findings. Convert high-risk bugs into tickets with clear remediation actions.

As you progress, track the costs of incidents avoided and efficiency gains. This evidence helps justify investment in an enterprise tool if you need it.

A contrarian view: why more testing is not always the answer

Not everyone will agree that red teaming should be routine. Some argue that heavy-handed testing slows product velocity and creates false negatives when teams chase exotic attacks at the expense of core user needs. There is merit to that view. Excessive focus on rare threats can divert resources from basic hardening measures like rate limiting, access control and encryption.

My stance is pragmatic: test what matters most. Prioritise scenarios that map to real-world attacker goals. Use simple, repeatable checks for the rest. In other words, be surgical with your efforts – plenty of value comes from a few well-chosen exercises rather than a scattergun approach.

When to consider an enterprise platform

Enterprise platforms make sense when you need scale, compliance and dedicated support. Consider them if:

• You operate in a regulated sector that requires audit trails and certified controls.
• Your volume of changes is high and you need automated, vendor-maintained regression suites.
• You lack in-house expertise and want a managed testing service with clear SLAs.

Even then, integrate enterprise tools with in-house practices. Platforms are most effective when used to amplify a mature testing culture, not to replace it.

What to do tomorrow: a short checklist for teams

If you take nothing else from Ravi’s story, do these four things tomorrow:

Run an end-to-end test that includes retrieval and any function calls your agent can make.

Add at least one multi-turn adversarial scenario to your regression tests.

Log conversation context and retrieval snippets so you can analyse attacks after they happen.

Hold a short tabletop with product, engineering and legal to agree on who acts when a risky behaviour is discovered.

These steps are small but powerful. They move teams away from a fragile one-off test and towards a habit of continuous improvement.

Closing thought: sensible scepticism wins

There is real hype around both open-source tools and paid platforms. Be sceptical of claims that any single solution will make your system invulnerable. The honest truth is that security is an ongoing process – one that combines creativity, testing and pragmatic engineering. Free tools can be highly effective when used intelligently. Enterprise platforms can speed adoption and provide organisational safeguards. The right choice depends on your risk profile and operational needs.

Ravi’s night was inconvenient and expensive, but it forced a useful change. The company moved from a checklist mentality to a living testing practice. As you build with models, remember that attackers are inventive. Make red teaming a deliberate, repeatable part of your work – and do not stop testing after the first success.