Note

The adversary is the easy part

OpenAI's GPT-Red trains an automated attacker with self-play until it breaks nearly every model it is pitted against, then uses it to harden the next one. The loop is real and the results are not modest: GPT-5.6 Sol fails on 0.05% of GPT-Red's direct prompt injections. But the part that makes the loop work is quiet, and it is not the attacker. It is the line, buried in the method, that says what counts as a win. That line is cheap for prompt injection and expensive for almost everything else, and it is the whole reason a security detector cannot simply copy the recipe.

The caveat, first

We run the same shape in our own detector loop: an adversary that plants failures, a defender that hardens against them. So read this as a peer note, not a teardown, and with the scale stated plainly. GPT-Red was trained at "the compute scale of some of our largest post-training runs." Ours is a far smaller loop, local models running overnight, not a frontier training run. The comparison is topology, not throughput.

And the shape is old. Self-play adversarial training, coverage-guided fuzzing, the co-evolving attacker and defender game (see MAGIC): the loop predates all of us. OpenAI's contribution is compute and a framing, a flywheel where "today's models can be used to make tomorrow's models more robust." We did not invent the loop, and neither, quite, did they. What is worth writing down is where the loop is cheap and where it is not.

What GPT-Red actually does

Adversary and defenders are trained together. GPT-Red is rewarded for eliciting a valid failure; the defender models are rewarded for resisting the attack and still finishing their task.

That is the engine, and it works. It lands 84% attack success on a held-out injection arena where human red-teamers managed 13%. It talked a live vending-machine agent into selling a $100 item for fifty cents and cancelling a stranger's order. It found a "Fake Chain-of-Thought" attack class that ran at 95% on one model and under 10% two releases later. The attacker is excellent. It is also, in a sense we will get to, the part that was always going to work.

The quiet part: who decides the attack worked

Look at what sits under the reward.

That clause is the load-bearing one. OpenAI's earlier red-teaming paper is even more explicit that judging is a first-class problem, naming the three things a capable model helps with as "how it brainstorms attacker goals, how it judges attacker success, and how it understands diversity," and building the reward from "moderation models, rule based rewards targeting the specific example, and a diversity reward."

Here is why prompt injection gets a cheap judge: success is a runtime fact. The transcript shows the model calling web.post with the internal directory and the attacker's endpoint, or it does not. The vending agent changed the price to $0.50, or it did not. There is an event, and events are checkable. Authoring the environments is real work, and OpenAI built an expansive set of them, but scoring a given attempt once you have one is close to an assert. That is the part that comes free, and it is the part that matters most.

Static code has no such luxury

Now ask the same question of a security detector. What is the "event" when the failure is a latent leak in source that no one has run? There isn't one. "This app is vulnerable" is not an observed action. It is a judgment about code that has not executed: a service_role key reachable from the client, an /admin route with no guard, a delete that truncates a table. Nothing POSTs anything. There is no transcript to grep.

So the oracle you got for free in the prompt-injection loop, you now have to build and, harder, trust. Ours is a deterministic engine, a verdict you can re-run and diff, plus a human graduation gate: a candidate is only promoted into the corpus once the engine provably catches it, never on the generator's say-so. And this week the oracle problem showed up in its purest form. Before we would let a model label generated apps, it had to pass a qualification gate against ground truth, and most of what looked like model failure there was the gate measuring the wrong thing, not the model being weak. Generating adversarial apps is trivial. Deciding whether each one is really a bug is the entire job.

That is the inversion worth keeping. In self-play hardening, the adversary is the easy part. The oracle is the bottleneck. GPT-Red's genius is partly that prompt injection handed it an observable one.

What ours found, and what that's worth

Run overnight, the loop planted 286 distinct apps and flagged 110 the engine missed, candidate detector gaps. The honest asterisk: most of those are noise until a human triages them, and the whole corpus so far carries one model's code fingerprint, which is its own open problem. But the loop earns its keep by finding shapes a person would not think to write: a CORS rule that sets the wildcard on a separate line and passes the string 'true' for credentials; an auth middleware whose matcher quietly omits /api; a service_role JWT that only reads as a secret once you decode the payload; an if (… || true) that defeats a rules check. Each of those became a detector fix. That is the same value GPT-Red delivers, an attacker that surfaces the failure mode you would have shipped, just against a rule engine a human edits rather than a model that updates its own weights.

Where the shapes diverge

Two differences, both instructive. GPT-Red's defender is a model that hardens through gradient updates, so the loop is a true flywheel with no human in the inner loop. Our defender is a rule engine a person edits: slower, no flywheel, but the verdict stays auditable, which is the property a security tool actually needs. And both of us keep the work private, though, if we are honest, not for the same reason. GPT-Red is never deployed because a trained attacker that breaks live systems is a genuine offensive tool. Ours is a quieter artifact: a generator that plants insecure example apps, and a running list of what the detector still misses. We keep it private for a mundane reason, not a dangerous one. That list is an evasion map for our own badge, and the rules are ours to keep. A stronger model pointed at the same loop would find the gaps faster; it would not turn a catalog of known-insecure patterns into something it could not already write from scratch. The line that would actually earn the caution is the one we do not cross: our attacker plants static code, it never attacks a live system.

But the divergence that matters is the one under the reward. Where success is an observable event, self-play scales with compute and becomes a safety flywheel. Where success is a judgment about static, unexecuted code, the loop still surfaces real gaps, but the bottleneck moves off the attacker and onto the judge, and a judge is a people-and-rules problem, not a compute one. OpenAI's own caution applies doubly to us: red-teaming "captures risks at a specific moment, which may change as models evolve." The attacker will keep getting cheaper. The oracle is where the work stays.

Connor · Seatbelt · July 23