Skip to main content
All writing

Testing4 min read

Your AI Agent Says the App Works. Here's the Proof It Doesn't.

A crashed React app returns HTTP 200 with an empty page. Every check that stops at the status code passes it.

Here is a component. It compiles cleanly. TypeScript is happy. The dev server reports no errors, and a request to the page comes back 200.

src/App.tsx
export default function App() {
  const user = undefined as unknown as { name: string };
  return <div>Hello {user.name}</div>;
}

It renders nothing. React throws on the first render, unmounts the tree, and leaves you with an empty root div inside a perfectly valid HTML document. The server is fine. The build is fine. The page is blank.

This is the single most common way an AI-built app is broken, and it is invisible to almost every check an AI builder runs on itself.

Three questions, only one of which matters

There is a ladder of things you can ask about a generated app, and each rung costs more than the one below it:

Does it compile?

Read the dev server's output. Catches syntax errors and bad imports. Cheap, fast, and passes the component above.

Do requests succeed?

Ask each route for a status code. Catches a server that is down or an endpoint that throws on import. Also passes the component above.

Does the page render?

Load it in a real browser and run the JavaScript. This is the only one that fails.

Most tooling stops at the second rung, because the second rung is easy and produces a green tick. The gap between rungs two and three is exactly the gap between "the agent said it was done" and "the user is looking at a white screen".

What a browser sees that a fetch does not

Run that same broken page through a headless Chromium and the difference is not subtle. The status code is still 200. Everything else is new information:

  • Whether anything rendered at all — a page with no text and no images is a white screen, however healthy its headers.
  • Uncaught exceptions, with the actual message: "Cannot read properties of undefined (reading 'name')".
  • Console errors, including React's own report of which component blew up.
  • Requests the page itself made and lost — the fetch to an API route that 500s on load.
  • A screenshot, which is the one artifact a person can judge in a second.

The signal-to-noise problem

A naive browser check reports a websocket error on every single run, because the dev server's hot-reload socket is configured for the public preview host and cannot be reached from inside the container. Ship that and everyone learns to ignore the output within a day. Filtering the errors that are structurally guaranteed is not cheating — it is the difference between a check people read and a check people mute.

"It compiles" is a claim about the code

The deeper issue is that an agent writing code has no feedback loop for the thing it is actually being asked to produce. It writes a file, the file is syntactically valid, and it reports success. It has never seen the page. It is describing its intent, not the result.

Give the agent a browser and the loop closes. It can open what it just built, read the errors, take a screenshot and look at it — and, more usefully, walk a flow: fill the email field, click Create account, expect the word "Welcome" to appear. A signup form that renders is not a signup form that works, and the difference is three clicks nobody was making.

A test that only proves the server answered is a test that passes on every white screen you will ever ship.

Where you run the browser matters

There is a tempting shortcut: run the headless browser on the machine that runs the agent. It is one less moving part, and the preview is only a port away.

It is also the machine holding your API keys, and the page it is about to load is full of JavaScript a language model wrote thirty seconds ago. If you have already decided that model-authored shell commands run inside a disposable container rather than on the host — and you should have — then model-authored JavaScript deserves the same answer. The browser belongs next to the app it is grading, in the sandbox, not next to the credentials.

It costs about a gigabyte of image size, which is the honest price of the feature. Chromium is not small.

Tell "broken" apart from "unavailable"

One last detail that decides whether people trust the check. If the browser cannot run — an old sandbox image, a disabled feature — the check must not report that as a failing app. "Your environment predates browser checks" and "your app renders nothing" look identical if you collapse them into one red mark, and the first one sends people hunting for a bug in code that is fine.

Isn't this just end-to-end testing?

It is a small, automatic slice of it. The difference is who it is for: an E2E suite is written by a person to protect a behaviour they care about, while this runs after every change to answer one question — does the thing I just built still render. It does not replace tests you write. It replaces the agent guessing.

Does it slow every change down?

It adds a few seconds after a turn already took a minute. And it only speaks up when something is wrong: a passing check that announces itself every time is a check people stop reading.

What about pages that are legitimately empty?

"Rendered nothing" has to mean nothing at all — no text and no images, canvases, inputs or buttons. A photo grid with no words is a real page. An empty root div is not.

If your AI builder tells you the app works, ask it what it looked at. If the answer is a status code, it does not know.

Keep reading