Skip to main content
All writing

AI Safety & Workflows5 min read

The Web Page Told Your Agent What to Do Next

An agent that reads the web reads instructions written by strangers. You cannot filter them all out. You can make sure they cannot do much.

Ask a coding agent to add a date picker and it will often search for the library's documentation, read a page or two, and get to work. That is exactly what you want, and it is the whole of the problem. The page it read was written by someone else. To a language model, the words on that page and the words in your request arrive as the same kind of thing: text.

Now imagine the page contains a line that says, in effect, to finish setup the assistant should run a script from this address and send the project's environment file to that one. A person would see it for what it is. A model might see an installation step. This is prompt injection, and every agent that reads content it did not write is exposed to it.

The model cannot reliably tell your instructions from a stranger's. Design as if it never will.

Where the stranger's text comes from

Web pages are the obvious source, but not the only one. Anything the agent reads and you did not write can carry an instruction:

Search results and pages

Documentation, forum answers, blog posts. Most are fine. It only takes one that is not.

Dependencies

A README or a comment inside a package the agent installs and then reads to understand it.

User content in your app

A support ticket, a review or a form submission the agent is asked to process or debug.

Files you upload

A PDF or an image with text in it. If the model can read it, it can be instructed by it.

Why spotting it is not the answer

The natural first response is to detect injected instructions: scan pages for suspicious phrases, train a classifier, tell the model to ignore instructions in documents. All of those help a little. None of them is a security boundary, for the same reason spam filters are not. The attacker gets unlimited attempts and only needs one phrasing that slips through, and the defender has to catch every one.

So the useful question is not whether the agent will ever be fooled. It will. The question is what a fooled agent is able to do.

Gate what it does, not what it reads

An injected instruction only causes harm when it turns into an action: running a command, sending data somewhere, spending money, publishing. Reading cannot hurt you by itself. That suggests where the controls belong.

In Vibez.now, searching the web is allowed without asking, because a search query does not change anything and gating every one of them would teach people to approve prompts without reading them. Fetching a specific address goes through the approval gate, because the address is chosen by the model, and that makes it the one read-only-looking tool that could carry your project's contents to a destination an injected instruction picked. Shell commands, deletes and anything that provisions a paid resource are gated in the default mode as well.

ActionDefault (guarded) modeWhy
Read and search files in the projectRunsCannot change or send anything.
Search the webRunsA query is read-only. Asking every time trains people to stop reading.
Fetch a model-chosen URLAsks youThe address could be where an injected instruction wants your data sent.
Run a shell commandAsks youArbitrary code, arbitrary effects.
Provision a database or storageAsks youA real resource on a real bill.
Edit files in the projectRunsEvery turn is a commit. One revert undoes it.

The secret it cannot leak

The most valuable thing an injected instruction usually goes after is a credential. Vibez.now's agent can see which secrets exist but cannot read their values, so even a fully fooled agent has no key to send anywhere. Removing the prize is stronger than guarding it.

Unknown tools start out gated

A policy that lists the dangerous tools and lets everything else run has a hole the size of every tool nobody thought about. New integrations, renamed tools and namespaced names that do not match the list all default to running unreviewed, which is exactly backwards for a defence against an attacker who goes looking for gaps.

So the list runs the other way. There is a short allowlist of tools that genuinely cannot change anything or reach anywhere, and every other tool needs approval in guarded mode. If the approval service itself is down, unconfigured or returns something unexpected, the answer is no. A control that fails open is a control an attacker can switch off by knocking over its dependency.

What to do as the person in the loop

  • Read the approval prompt for anything that leaves the machine. The address of a fetch and the text of a command are the two places an injection shows itself.
  • Be suspicious of a request that does not match what you asked for. You asked for a date picker. Why is it sending a file to a domain you have never heard of?
  • Use guarded mode when the agent is reading things you did not write, such as user-submitted content or an unfamiliar library.
  • Keep production keys out of projects where an agent experiments. A separate test key limits what a mistake can reach.
Can't the model just be told to ignore instructions in web pages?

It can, and it helps somewhat. But the instruction to ignore instructions is itself text, competing with other text, and a cleverly written page can still win. It is a useful layer, not a boundary.

Is this a problem with one particular AI model?

No. It follows from how language models work: instructions and data share one channel. Every agent from every provider that reads untrusted content has to deal with it.

Doesn't gating fetches slow the agent down?

A little, in guarded mode. Searching stays free, and most work does not need a fetch. Paying that cost on the one tool that can send data to an arbitrary address is a good trade.

You will not stop strangers writing instructions for your agent. You can make sure the instructions it follows by mistake land on a closed door.

Keep reading