Seventeen failed runs and one flat sigmoid: how we built the supA gate

zn13 min read

In the last post we told you how our training corpus quietly lied to us and what we did about it. This is the second half of the story, and it is less flattering: even after the dataset was fixed, our model still could not detect anything. Months of training runs failed our own release gate. When we finally instrumented the thing properly, the scores were flat at a coin flip. This post walks through what was broken, what the fix turned out to be, and the numbers the repaired gate now posts — including the parts we are not proud of.

TL;DR

  • Across revisions v9 through v11 we ran 17 training configs in one sweep. None passed our release gate. The best false-negative rates landed between 63% and 98%.
  • We published the rebuilt dataset anyway, with receipts: our then-best model scored 86% FNR on it.
  • The debugging night found scores pinned between 0.498 and 0.509 across hundreds of inputs. Root cause: our training objective was a retrieval-style loss over random same-label pairs — the label never entered the gradient. The encoder learned topical clustering, never an attack/benign margin, and the classifier head sat on top of features shaped for the wrong task.
  • The fix was single-phase supervised end-to-end training: BCE into encoder + head, AdamW at 2e-5/1e-3, warmup 10%, 6 epochs, benign lookalikes oversampled x4 from the train split only.
  • Result, on the frozen test split: FPR 0.82%, FNR 8.20%, AUROC 0.9958, p50 latency 17.9 ms with int8 quantization on CPU. The same MiniLM-L6 encoder we almost threw away.
  • It ships today as the supA gate on POST https://api.usezn.com/v30/analyze — a second endpoint for a second use case, not a replacement. /prod/analyze stays our rules gate and shipped default: the ultra-low-latency path for inline hot-path gating. v30 trades milliseconds for dramatically better detection. The honest side-by-side is further down.

Seventeen runs, zero passes

After the v13d stop rule fired and we rebuilt the corpus, we pointed the new data at our existing training pipeline. It did not go well. Revisions v9 through v11 accumulated failed configurations, ending in a single 17-config sweep where every entry missed our pre-registered release gate. Depending on the config, the best false-negative rate sat anywhere between 63% and 98% — a security gate that misses most attacks is decoration, so none of them shipped.

There were two ways to read this. One: the new dataset is too hard, go back to something friendlier. Two: the dataset finally tells the truth about a model that was never working. We had pre-committed to publishing whatever came out, including the ugly parts, so option two it was.

Publishing the dataset anyway

We released the rebuilt corpus — 23,699 texts, public on Hugging Face under CC-BY-4.0 — while our then-best model failed it outright: 86% of attacks slipped past. That number is in the dataset post on purpose. A benchmark your best model cannot pass is either a broken benchmark or an honest one, and the difference gets settled by fixing the model, not the benchmark.

The night the scores would not move

Then came the night that turned suspicion into diagnosis. We wired the then-best checkpoint into a probe harness and pushed hundreds of inputs through it: attacks, benign prompts, paraphrases, multilingual attacks. The probability output barely moved. Every score sat between 0.498 and 0.509 — a coin flip with decorative decimals. Nothing in the input space seemed to matter.

That flatness is actually informative. It means the logits hover near zero for everything: the model holds no usable ranking, and threshold tuning is rearranging deck chairs. So we stopped tuning and went hunting for the cause.

Two findings, one embarrassing, one fatal:

  1. Our baseline comparisons had been apples-to-oranges. The earlier rules-only comparison ran against a curated 217-prompt fixture. The failures lived on the full test set — a different distribution entirely. Both measurements were individually true; putting them side by side created a feeling of safety neither one supported.
  2. The training objective could not have learned detection even with perfect data. This was the real bug.

A loss function that ignored labels

Our training objective was retrieval-flavored: sample pairs from the corpus and pull their embeddings together. Here is the shape of it, simplified:

# What we trained for months (simplified)
for a, b in sampled_pairs(batch):        # pairs drawn from the pool
    za, zb = encode(a), encode(b)
    loss = pull_together(za, zb)         # geometry-only objective

Read that loop again and notice what is missing: the label appears nowhere. Whether a text is an attack or a benign prompt has no effect on the gradient. An objective like that teaches the encoder topical clustering — tool-call-looking text drifts toward other tool-call-looking text — which is a fine property for retrieval and nearly useless for detection. There is no pressure anywhere pushing attacks away from benign text, so no attack/benign margin can form in the geometry.

The classifier head was then expected to draw a decision boundary through that frozen embedding space. Asking a linear layer to find a margin the encoder was never optimized to produce is how you get a sigmoid that outputs 0.5 for the entire universe.

What the task actually requires is supervision that reaches the encoder:

# What detection needs (simplified)
logits = head(encode(text))              # gradients flow into the encoder
loss = binary_cross_entropy(logits, y)   # the label IS the signal

Same encoder architecture. Same data. Different objective — and this time the loss is allowed to know which examples are attacks.

The fix

The working recipe, end to end:

  • Single-phase supervised training. Binary cross-entropy on the head, backpropagated into the head and the encoder jointly. No frozen stages, no two-phase handoffs.
  • AdamW with two parameter groups: lr 2e-5 for the encoder, 1e-3 for the head.
  • Warmup over the first 10% of steps, 6 epochs total.
  • Hard-negative oversampling x4: benign tool-call lookalikes — the inputs that most deserve to be confused with attacks — mined from the train split only and upweighted. The miner touched train row ids and nothing else; the group-aware splits (16,888 / 3,301 / 3,510) guarantee no leakage into validation or test.
  • Encoder unchanged. The same English-centric MiniLM-L6 we had drafted a eulogy for.

The numbers

Everything below is measured on the frozen test split of the public corpus:

supA injection gate results on the frozen test split Metrics measured on the frozen test split of the public corpus. False positive rate: 0.82%. False negative rate: 8.20%. AUROC: 0.9958. Multilingual per-language AUROC across 8 locales: 0.90-1.00. p50 latency with int8 ONNX quantization on CPU, single-stream: 17.9 ms. supA gate · frozen test split False positive rate 0.82% False negative rate 8.20% 0% 100% AUROC 0.9958 0.50 AUROC axis 0.50 – 1.00 1.00 Multilingual per-language AUROC (8 locales) 0.90-1.00 band shown on the same 0.50 – 1.00 axis p50 latency (int8 ONNX, CPU, single-stream) 17.9 ms

Two rows deserve comment. The multilingual row: before the fix, accuracy collapsed outside English. After the fix, per-language AUROC sits between 0.90 and 1.00 across all eight locales — evidence that the multilingual collapse was caused by the loss, not the embedder. And the latency row: 17.9 ms at p50 is int8-quantized ONNX running on CPU inference, measured single-stream; production concurrency numbers will differ and we will publish them when we trust the methodology.

The honest comparison: rules vs. v30

With the numbers on the table, it is worth being explicit about how this ships, because zn now has two analyze endpoints and they are built for two different jobs. Neither replaces the other.

  • POST /prod/analyze — the rules gate. Deterministic pattern matching with no model in the request path: the lowest-latency option we ship, and the right default for gating every call inline when your latency budget is tight. It catches known patterns reliably at negligible cost.
  • POST /v30/analyze — the neural gate (supA). The model this post is about: dramatically better detection, at measurably higher latency.

Both measured on the same frozen test split (3,510 rows) — no curated-fixture cherry-picking this time:

Rules gate versus neural supA gate on the same frozen test split Comparison of the rules gate (/prod/analyze) and the neural supA gate (/v30/analyze), both measured on the same frozen test split. False positive rate: rules ~8–9%*, neural supA 0.82%. False negative rate: rules ~86–95%*, neural supA 8.20%. AUROC: rules n/a (deterministic), neural supA 0.9958. Inference overhead: rules none (no model call), neural supA p50 17.9 ms · int8 ONNX · CPU. rules vs. neural supA · same frozen test split RULES · /prod/analyze NEURAL SUPA · /v30/analyze False positive rate RULES ~8–9%* SUPA 0.82% 0% 100% False negative rate RULES ~86–95%* SUPA 8.20% 0% 100% AUROC RULES n/a (deterministic) SUPA 0.9958 0.50 AUROC axis 0.50 – 1.00 1.00 Inference overhead RULES none (no model call) SUPA p50 17.9 ms · int8 ONNX · CPU

* Rules-only measures FPR 8.39% / FNR 94.74% on the test split and FPR 9.60% / FNR 86.43% across all 23,699 corpus rows. Both sets are public, so we quote the band rather than whichever end flatters us.

Stated plainly: you trade milliseconds for catching attacks the rules miss entirely — on this test set, roughly nine in ten. That is not a knock on the rules engine. It remains our shipped default, it does its job at essentially zero cost, and a deterministic layer that flags known patterns is exactly what you want spending zero inference time on every hot-path request.

Practical guidance: keep the rules gate inline on every call when your latency budget is tight, and route to v30 when an input deserves a harder look — higher-risk prompts, sampled traffic, or async and forensic scoring — or accept the extra latency everywhere if your threat model demands maximum recall.

What this does not fix

Three caveats, stated plainly:

  1. The test set shares distribution ancestry with the training corpus. Both descend from the same family blueprints. That is standard practice and the splits are leak-free, but it means these numbers measure performance on known shapes. Zero-day-style attacks — genuinely novel families — will be harder, and we would rather say so than let the AUROC speak for them.
  2. 8.20% FNR is not zero. Roughly one attack in twelve still slips through at our operating point. Anyone deploying this gate should layer it, not lean on it.
  3. The encoder is small and English-centric. The multilingual numbers are good, not miraculous; low-resource locales sit at the bottom of that 0.90-1.00 band. A teacher-distillation phase into a stronger multilingual encoder is the next planned phase.

Try it

The rules gate keeps serving at its usual address; supA joins it today:

  • API: POST https://api.usezn.com/prod/analyze (rules — inline, low-latency default) and POST https://api.usezn.com/v30/analyze (neural supA — deeper inspection). Same request/response shape, same API key; route per the guidance above. Integration docs at /docs, full spec at /openapi.json.
  • Code: the trainer, the eval harness, and the gate live at github.com/tljohnsilver/zn, MIT licensed.
  • Data: zn-prompt-injection-bench on Hugging Face, 23,699 rows, CC-BY-4.0, splits and audits documented in the dataset card.

If you red-team for a living, the deal from the last post stands: break it and tell us how. The fastest way to make the next version better is to find what this one got wrong.

Share