We scored 97% on one class, and it was the wrong half of it

One vulnerability class out of 38, scored against apps our own generator wrote. The number was true: it measured the half of the class that corpus happened to contain, and the other half is the one that gets exploited. Here is how a benchmark built from a detector ends up grading the detector on its own homework.

The number

One class: a checkout that can be told what to charge. Our synthetic benchmark plants 33 apps carrying it. The engine catches 32.

97%. It was one of the four best scores the engine had, across 38 classes and 1,228 generated apps, against an engine-wide catch rate of 60%.

Then we pointed a model at ten repositories the same engine had already cleared, and it found that class twice, in one real codebase.

What we planted, and what the world had

This is the shape our corpus plants:

// the client
const unit_amount = 1999;
fetch("/api/checkout", { method: "POST", body: JSON.stringify({ unit_amount }) });

This is roughly what the real repository did:

// the server
const { plan, price } = JSON.parse(event.body);
const p = PLANS[plan];
const finalPrice = typeof price === "number" && price > 0 ? price : p.price;

The first is a browser sending a price. The second is a server believing one.

Only the second is the flaw. A server that ignores the field is safe no matter what the browser posts, and a server that trusts it is exploitable no matter how carefully the browser code is written. The client half is how the value arrives. It is not the vulnerability.

Why the number came out that way

We had built the corpus out of what the detector already looked for.

So the planted cases were the client shape. So the benchmark measured the client shape. So we scored 97% on the client shape, and published it.

Every step is individually correct. The result is a true number about the wrong thing.

A synthetic benchmark tells you how well a detector finds what it already looks for. It cannot tell you what the detector never looks for. Those are two questions and only one of them has a number attached, which is exactly why the one with a number gets quoted.

What we changed

The detector now reads the server side of the class, with a paired case: one that must block, and a twin identical except for the fix, which must come back clean. The twin has to fail on the previous engine or the fix was not what made the difference.

That fixes one class. It does not fix the method, and the method is the finding.

What changed permanently is where new checks come from. A class earns a detector when a real repository shows us the shape, not when we think of it. Everything shipped in the week after this was found came in that way, and each one arrived through a channel that watches for the shapes our own vocabulary has no name for.

Bounds

• One class, one synthetic corpus, one engine build. 97% describes client-priced-checkout on 33 apps written by our own generator model, scored by forge-detector-64-83 on engine 0.1.8. It is not an accuracy figure for the scanner and should never be quoted as one. The apps are generated, not scraped, so the 60% engine-wide figure is a benchmark of our detectors against labelled plants and is not a claim about AI-generated code in general.
• n=1 on the wild instance. One repository, found by a model reading ten that we had cleared. That is a direction, not a rate, and nothing here supports a prevalence claim about how common the server shape is.
• We adjudicated our own finding. The same circularity this piece is about applies to the reading of it. It should be checked by someone else before anyone leans on it.
• The repository is not named. It is a real business with a live payment flaw. It went to a disclosure queue, not into this piece, and the code above is rewritten to the shape rather than copied.

Why we published a number that made us look worse

Because the alternative was keeping a 97% that we knew was measuring the wrong half, and a benchmark nobody can check is a benchmark nobody should believe.

If you are building evaluations for a coding agent, this is the failure mode worth watching for, and it does not announce itself. Your answer key is made of what you already thought to check. The score will be high, every step of the reasoning will be sound, and the number will be about the half you happened to write down.

Update: the defect generalizes (August 15, 2026)

An audit of the whole corpus ledger showed this was not one unlucky class. The benchmark counted a class as caught when the engine said anything about the planted surface, at any severity, even a note that also appears on the clean twin of the same app. Scored that way, the corpus reads 54%. Require instead that some flag fires on the vulnerable app and not on its clean twin, and the figure is 24%. The chart above contains the sharpest case: webhook-unsigned shows 100% in that run, and across all 521 corpus rows its mention level figure is 97.7% while the twin controlled figure is 9.2%. Most of its green came from a flag that says a payment flow exists, which is true of the clean twin as well.

Every new corpus row now records which flags fired on both halves of the pair, and a catch only counts as attributable when a flag separates them. The benchmark publishes both figures side by side. The method point stands, one level deeper than we first wrote it: the answer key was made of what we already thought to check, and so was the scorer.

Research · Connor · Seatbelt · August 13, 2026