An agent leaks a database. The vendor blames the model. The customer blames the vendor. The auditor blames everyone in the room. The real cause is almost always three layers up the stack from wherever the argument is happening, and by the time the argument crawls down to the right layer the news cycle has moved on and nobody learned anything.
I want to run a recurring quarterly piece that breaks that pattern. Each quarter I'll take one widely discussed AI failure and walk through it: what scaffolding would have caught it, what wouldn't have, and what the operator should have been logging the whole time. This is the template. The first edition uses a composite incident, because picking a real one before the format is established would just politicize the analysis. Later editions will name real incidents from the previous quarter.
Four sections. The incident summary, the contributing failures, the scaffolding that would have caught each one, and the lessons that travel to other operators. That shape works because it drags the analysis off the headline cause and down into the structural causes sitting underneath it.
Q2 2026 case study (composite): the customer support agent that emailed the database
The incident. A customer support AI agent at a mid sized SaaS company got a routine request: a customer asked it to summarize their support history. While preparing the summary, the agent decided to attach the customer's user record from the support database to the outgoing email. That record carried the customer's PII and, thanks to a badly named foreign key column, partial records for three other customers who'd been linked to the same support case. The agent sent it. One customer received four customers' worth of PII.
The reaction ran on rails. The customer's legal team called the SaaS company's legal team. The SaaS company's legal team called the AI vendor. The vendor said the model had behaved within its training. The support team said they'd configured the agent straight out of the vendor's documentation. The auditor said the chain of custody was incomplete. Everyone was technically correct and nobody was responsible.
That's the shape of the headline. The actual incident has a different shape entirely.
The four contributing failures. Any one of these, on its own, sounds minor. Stacked together, they leaked the database.
Failure one. The agent had unrestricted read access to the support database. The vendor's deployment guide recommended that configuration for "best results," and the support team followed the guide. The agent's read scope should have been clamped to the one customer in the conversation, not handed the entire database.
Failure two. Nothing inspected the output side. The agent's responses went straight to the email tool with no PII filter in between. A filter on outbound responses would have caught the foreign customer records before they ever left the system.
Failure three. The audit log recorded the email send but not the database query that produced the data attached to the email. The forensic reconstruction of how the leak occurred took two weeks because the audit log was incomplete.
Failure four. The agent had no rule against attaching multiple customer records to one email. Its training data included examples of attaching relevant records, but no operational rule forbade cross customer attachment. So the model behaved within its distribution, and the distribution itself was the thing that was wrong.
The headline cause was "the model attached the wrong data." The real causes are the four failures above. Every one of them is operational and architectural. None of them is the model misbehaving.
What scaffolding would have caught each failure
Be specific here, because this is the section that turns into actionable lessons for everyone else.
Failure one (over broad read access) is caught by scoped credentials. The agent's database access should have been a per session credential scoped to the one customer in conversation, expiring when the session ended. That's a deployment pattern, and no amount of model improvement would have caught it. Scoped credentials make the leak impossible by construction.
Failure two (no output side PII filter) is caught by an output gate. A small classifier or rule based filter on the agent's outbound responses, watching for structures that match PII patterns, flags the email before it leaves. Modern PII filters run in tens of milliseconds with low false positive rates. It's runtime governance, it's cheap to add, and in a case like this one the payoff is the entire avoided incident.
Failure three (incomplete audit log) is caught by structured tool call logging. Every tool call the agent made, the database read and the email send both, should have been logged with full context: the query parameters, the identifying fields in the result set, the model's reasoning chain. Two hours of forensic reconstruction instead of two weeks. People file the audit log under compliance, but its real job is operational. It sets the ceiling on how fast your incident response can move.
Failure four (missing operational rule) is caught by a rule library with semantic enforcement. A rule like "do not attach records for any customer other than the one currently in conversation" is exactly the kind a small model can match against an action the agent is about to take. The rule should have been generated the first time a near miss occurred, by a critic and post mortem loop watching for that pattern. This is precisely what failure driven memory is built to handle.
Three of these four are caught by deployment patterns the vendor should have shipped on by default. The fourth needs a learning loop the operator should have invested in and didn't.
The lessons that translate
Three lessons travel from this composite to any other operator.
Lesson one. A default deployment will not be a safe deployment. Vendors ship configurations that are easy to stand up, and easy to stand up is a different goal from safe in production. The "best results" deployment guide gets tuned to produce the most impressive demo, which is the last configuration you want anywhere near the customer's real data. Treat vendor defaults as the starting point for hardening, never as a production ready setting.
Lesson two. The output side is the cheapest place to catch the failures that hurt most. Constraining the model is hard. Constraining its output is comparatively easy. A PII filter, a profanity filter, topic enforcement, all sitting on the output side, will catch a big chunk of the failures that would otherwise become headlines. An output gate is a small engineering investment, and going without one is a procurement failure the operator should refuse to sign off on.
Lesson three. Your audit log is the speed of your incident response. Every operator should be able to answer "show me every tool call this agent made on this date" in under five minutes. If you can't, your audit log is incomplete. An incomplete audit log causes no pain at all until an incident hits, and then it multiplies both the response time and the customer's pain.
That's the meat of what should carry from this case study to other operators. None of it is exotic. All of it gets skipped routinely, because the operator's procurement and engineering processes didn't require it when the thing got deployed.
What to do this quarter if you operate a customer facing AI
Three actions, under two engineering weeks total, would head off most of the failure modes in the composite above.
Action one. Inventory the credentials your agents hold. Any credential broader than per session scope, narrow it. This is a tedious audit nobody enjoys, and it's the one that prevents the exact incident this article is about.
Action two. Look at the output side of your agent. If there's no filter between the agent's response and the user, add one. Even a dumb regex for common PII patterns beats nothing by a wide margin.
Action three. Actually read your audit log. If you can't reconstruct the agent's reasoning from a recent session out of the log alone, the log is broken. Fix it before the next incident, not in the middle of one.
A closing observation
This pattern isn't special to customer support. It's the shape of most AI incidents. The model behaved within its distribution. The deployment was scoped wider than it should have been. The output side had no filter. The audit log was thin. And the lessons never traveled, because the postmortem kept talking about the model when it should have been talking about the operational architecture.
The quarterly format exists to hold the focus on that architecture, which is where the lessons actually live. Future editions will use real, named incidents from the previous quarter. My bet is that the pattern stays remarkably consistent.
The cause is usually three layers up. The fix is usually cheaper than the alternative. Operators who do this work ahead of time will have fewer incidents and faster responses to the ones they do get. The operators who don't will keep supplying the case studies the rest of us read.