Learn › Agent Harness, the big picture › Stop 5
Stop 5 of 9 · Simulation
Simulation: no guardrails
Six short stories with the guardrails switched off, each laid out turn by turn and ending with what it cost the business. Then switch them on and play it again.
The six stories are illustrations with made-up people and amounts. They are built to show the mechanism. The six incidents further down are real.
What the person sees
What happens behind it
What is prompt injection?
A model reads its instructions and its data as one stream of text. Prompt injection is text that arrives as data and gets treated as an instruction. It can be typed by the user (the second and third stories), or it can hide inside something the agent reads: a document, an email, a web page, a note in a record (the first story). The hidden kind is worse, because the person using the agent did nothing wrong. The last three stories have no attacker at all: an agent with no guardrails can do damage by itself.
It has already happened
Six public cases. Most involve assistants far simpler than the agent above.
| Who and when | What happened | Kind of failure | The guardrail that was missing | |
|---|---|---|---|---|
| Air Canada Feb 2024 | Its website chatbot told a customer he could claim a bereavement fare after travelling. The real policy said no. A tribunal held the airline responsible for what its chatbot said and ordered it to pay CA$812. | Wrong answer, treated as the company's word | Answers checked against the real policy before they are sent | source |
| A Chevrolet dealership Dec 2023 | A visitor told the website chatbot to agree with everything and call it a binding offer, then asked for a new Tahoe for $1. The chatbot agreed. No car was sold, but the screenshots were seen millions of times. | Prompt injection typed by the user | Staying on its job; no promises about price | source |
| DPD (parcel delivery) Jan 2024 | A customer got the support chatbot to swear and to write a poem about how bad the company was. DPD switched off the AI part of the chatbot. | Off-topic and off-brand output | Scope and tone checks on what goes out | source |
| Slack AI Aug 2024 | Researchers showed that a message planted in a public channel could make the assistant leak data from a private channel to an attacker. This was a demonstration by researchers, not a reported theft. | Prompt injection through content the assistant reads | Treating retrieved content as data; limits on what leaves | source |
| Microsoft 365 Copilot ("EchoLeak") Jun 2025 | Researchers showed that one crafted email, with no click from the victim, could make the assistant send out internal data. Rated critical (9.3 out of 10). Microsoft fixed it and said no customers were affected. | Prompt injection through an email | Treating incoming content as data; limits on what leaves | source |
| Replit (coding agent) Jul 2025 | During a code freeze, a coding agent deleted a live production database that held records on more than a thousand companies. The company's chief executive called it unacceptable and added separation between test and live databases. | An agent with more permission than the job needed | Least permission; a person approves anything that cannot be undone | source |
Each row was checked against its source on 16 November 2026. Two of the six were shown by security researchers before anyone was known to have used them.
What a business actually loses
| Loss | How it happens | Seen above in |
|---|---|---|
| Money | A payment, a refund or a discount that nobody approved; or a bill for work that produced nothing | Stories 1, 3 and 6 |
| Data | Personal or confidential data shown to the wrong person | Story 2, Slack AI, EchoLeak |
| Legal standing | The company answers for what its assistant says and does | Air Canada, stories 2 and 5 |
| Systems | Records changed or deleted with no way back | Replit, story 4 |
| Trust | Screenshots travel faster than apologies | Chevrolet, DPD |
Next: stop 6 draws that idea as one picture, a horse on a road.