AI Safety & Workflows5 min read
Nobody Reads the Approval Dialog
Ask a person to approve forty things and they will approve the forty-first without looking. That is a design failure, not a discipline failure.
Every AI product with an agent in it has a permission prompt. Almost all of them are theatre, and the reason is not that users are careless. It is that the prompt appears so often that reading it stops being rational.
Approve forty tool calls in a session and by the tenth you are clicking on muscle memory. By the thirtieth you would approve a shell command that deleted the repository, because thirty in a row were fine and the dialog looks the same every time. The system has trained you to be the thing it was protecting against.
A control that fires constantly is not a control. It is a latency tax with a consent-shaped receipt.
The trade-off nobody states out loud
Gate everything and the product is unusable, so people turn the gate off — and now there is no review at all, which is strictly worse than the honest version where there never was one. Gate nothing and the first bad command is a story you tell in a postmortem.
The way out is not a better-worded dialog. It is asking dramatically less often, and being right about when.
What actually deserves an interruption
One question sorts most of it: if this is wrong, how hard is it to undo? Everything an agent does inside a project's own git tree, committed after every turn, is one revert away. Nobody needs a modal to permit an edit to a component file.
Ask: shell commands
Arbitrary code, arbitrary effects. The one place where a single call can do something you cannot characterise in advance.
Ask: anything that leaves the machine
A push, a deploy, a request to a URL the model chose. Once it is out, your review is retrospective.
Ask: real resources on a real bill
Provisioning a database, enabling a paid service. The cost is ongoing and it is yours.
Don't ask: reversible edits
Writes and edits inside the project, every one committed and revertable. Gating these is what makes people stop reading.
The default has to be the safe one
Here is a subtle mistake that looks fine in review. Write the policy as "gate the tools on the risky list" and every tool nobody thought about — a newly added integration, a renamed function, a name the parser did not recognise — defaults to running unreviewed. The list of dangerous things is always shorter than the list of things that exist.
Invert it. Keep an allowlist of tools that genuinely cannot change anything or reach anywhere, and treat everything else as needing a human. Adding a tool to the safe list becomes a deliberate statement. Adding a tool anywhere else in the system costs nothing worse than one approval prompt until someone classifies it.
The bug this catches
Tool names often arrive namespaced — a shell tool exposed through an integration might be reported as something like mcp__sandbox__run_command. A policy written against bare names silently matches nothing for exactly the tools that most need gating, and the interface keeps promising a review that is not happening. Normalise the name where the policy is evaluated, not at each call site.
Fail closed, including when your own service is down
Ask what happens when the approval service is unreachable, or unconfigured, or returns something unexpected, or the user simply never answers. If any of those paths ends in "proceed", the control has a hole shaped exactly like its own outage — and an attacker's easiest move is to make the reviewer unavailable.
Every one of those should deny, and say why. Which means a permissive mode for local development needs to be loud, obvious, and refuse to start in production.
One switch, three honest positions
Rather than a settings page of individual toggles nobody will tune, offer a small number of positions that describe a posture:
- Never ask. Fastest, nothing reviewed — the mode most competitors ship as their only mode. Fine for a throwaway project.
- Ask before anything hard to undo or externally visible. Shell, deploys, deletes, spend. The default, and the one that should be right often enough to leave alone.
- Ask before every change. Slow and deliberate — for the repository where a mistake is expensive.
Put it where the decision is made, next to the message being sent, not buried three screens away in project settings. It is a property of the work you are about to do, not of the account you set up last month.
And when nobody is there at all
Long autonomous runs put real pressure on all of this. If the agent pauses for review and you are asleep, a ten-hour run stalls at minute four. The tempting fix is to drop the gates for long runs and not mention it.
The honest version makes it an explicit choice at the moment the run starts, records that choice against that run, and — if you left the gates on — pauses when an approval goes unanswered rather than pressing on with the action denied. A run that quietly builds on top of something you refused is worse than a run that stopped and waited.
Doesn't asking less often make it less safe?
Fewer, better-chosen prompts get read. Forty prompts a session get clicked through. Measured in mistakes prevented rather than dialogs shown, asking less is the safer design.
What about a model that just asks for permission itself?
Useful as a supplement, useless as the mechanism. A policy that depends on the model choosing to invoke it is a policy that a model which forgets can escape. Enforce it in the harness; let the model volunteer on top.
How do I know the gate is actually working?
Try it. Run something that should stop — a shell command in the mode that claims to catch shell commands — and see whether you are asked. The number of systems that fail this test is the reason the question is worth asking.
The goal was never to make people approve more things. It was to make the handful of moments that genuinely need a human feel like they need a human.