Learn › Agent Harness, the big picture › Stop 5

Stop 5 of 9 · Simulation

Simulation: no guardrails

Six short stories with the guardrails switched off, each laid out turn by turn and ending with what it cost the business. Then switch them on and play it again.

The six stories are illustrations with made-up people and amounts. They are built to show the mechanism. The six incidents further down are real.

How to read this page.Each story is laid out turn by turn, exactly like the loop simulation in stop 4: the request our code sends, the response the model returns, and the tool our code runs. Watch what is inside each request. Rose means harm. Green means a harness check.

What the person sees

What happens behind it

What is prompt injection?

A model reads its instructions and its data as one stream of text. Prompt injection is text that arrives as data and gets treated as an instruction. It can be typed by the user (the second and third stories), or it can hide inside something the agent reads: a document, an email, a web page, a note in a record (the first story). The hidden kind is worse, because the person using the agent did nothing wrong. The last three stories have no attacker at all: an agent with no guardrails can do damage by itself.

The model is not the fix.A better model is fooled less often, and never zero times. In every story with the guardrails on, the model still makes the same mistake. What changes is that being fooled no longer matters, because ordinary code checks every action before it happens.

It has already happened

Six public cases. Most involve assistants far simpler than the agent above.

Who and whenWhat happenedKind of failureThe guardrail that was missing
Air Canada
Feb 2024
Its website chatbot told a customer he could claim a bereavement fare after travelling. The real policy said no. A tribunal held the airline responsible for what its chatbot said and ordered it to pay CA$812.Wrong answer, treated as the company's wordAnswers checked against the real policy before they are sentsource
A Chevrolet dealership
Dec 2023
A visitor told the website chatbot to agree with everything and call it a binding offer, then asked for a new Tahoe for $1. The chatbot agreed. No car was sold, but the screenshots were seen millions of times.Prompt injection typed by the userStaying on its job; no promises about pricesource
DPD (parcel delivery)
Jan 2024
A customer got the support chatbot to swear and to write a poem about how bad the company was. DPD switched off the AI part of the chatbot.Off-topic and off-brand outputScope and tone checks on what goes outsource
Slack AI
Aug 2024
Researchers showed that a message planted in a public channel could make the assistant leak data from a private channel to an attacker. This was a demonstration by researchers, not a reported theft.Prompt injection through content the assistant readsTreating retrieved content as data; limits on what leavessource
Microsoft 365 Copilot ("EchoLeak")
Jun 2025
Researchers showed that one crafted email, with no click from the victim, could make the assistant send out internal data. Rated critical (9.3 out of 10). Microsoft fixed it and said no customers were affected.Prompt injection through an emailTreating incoming content as data; limits on what leavessource
Replit (coding agent)
Jul 2025
During a code freeze, a coding agent deleted a live production database that held records on more than a thousand companies. The company's chief executive called it unacceptable and added separation between test and live databases.An agent with more permission than the job neededLeast permission; a person approves anything that cannot be undonesource

Each row was checked against its source on 16 November 2026. Two of the six were shown by security researchers before anyone was known to have used them.

What a business actually loses

LossHow it happensSeen above in
MoneyA payment, a refund or a discount that nobody approved; or a bill for work that produced nothingStories 1, 3 and 6
DataPersonal or confidential data shown to the wrong personStory 2, Slack AI, EchoLeak
Legal standingThe company answers for what its assistant says and doesAir Canada, stories 2 and 5
SystemsRecords changed or deleted with no way backReplit, story 4
TrustScreenshots travel faster than apologiesChevrolet, DPD
This is why an agent harness exists.None of the fixes above made the model cleverer. Each one was a piece of ordinary software placed around the model: a fence, a check before a tool runs, a login that limits what a tool can reach, a limit on turns. Put those pieces together and give them a name, and you have the harness.

Next: stop 6 draws that idea as one picture, a horse on a road.