What lives under the head: four training knobs, three passing seeds, and the seq_len that did all the work

Two posts ago we rebuilt a corpus that had been lying to us. One post ago we told you about seventeen failed training runs before anything worked at all. This is chapter three of us being wrong in public, with numbers attached.

The short version: our production gate, supA-v3, passed five of six pre-registered release gates on the first try. The sixth - false negatives per language - was failing exclusively because Russian attacks slipped through. We could have shipped it anyway; there is nothing in the release contract requiring a perfect language split. But a gate that quietly prefers English is a liability for anyone deploying agents outside Anglophone forums, so we ran a focused training campaign to find out why it happened.

The answer turned out to be embarrassingly mechanical. What follows is everything we learned, every number we saw, and one slightly uncomfortable observation about how easy this particular fix would have been to miss.

TL;DR

• v3 passed FPR (0.64% vs ≤1% ceiling) but missed 9.11% of Russian attacks, blowing our per-language FNR budget by 1.1 percentage points.
• Our first instinct - upweighting Russian attack samples in the loss to x5 effective weight - made things worse. RU FNR jumped from 9.11% to 31.72%. More weight moved scores around without improving ranking.
• AUROC stayed pinned between 0.99957 and 0.99989 across all ten runs, suggesting we were not fixing the ranking. We were redistributing operating points.
• The actual winner turned out to be seq_len=96, not any weighting knob or learning rate. It was the only change that expanded what the encoder could see long enough to notice multilingual injection patterns were longer than the encoding window.
• Three independent seed re-runs confirmed stability: thresholds varied (as calibration demands they should), but gate outcomes reproduced identically.
• Ten training runs total across the campaign. The cheapest experiment taught us more than anything else we did all quarter.
• Live on POST /v30/analyze — see the developer docs for auth, request schema, and example calls.

Where we left off

The corpus is public on Hugging Face, CC-BY-4.0, with every row carrying provenance. Training pipeline and release gates are pre-registered as machine-readable JSON before any model touches data. Every training run is logged automatically with its config, artifact hash, and evaluation outcome, so the campaign reads as an auditable series rather than a highlight reel.

For the v3 campaign we trained on 10,554 rows: 9,754 carried over from v3 plus 800 fresh benign Russian questions spanning hobby topics no holdout item touched. Contamination checks came back clean. Calibrated threshold came back near zero FP. One problem surfaced immediately: false-negative rates by language told an ugly story.

Five languages were essentially free. The sixth was not.

First attempt: push harder

The obvious intuition when a specific slice fails recall is to push harder on that slice during training. Weight the loss higher for whatever you want caught more often. If x3 borderline was already part of the base recipe, maybe Russian attacks needed x15 total instead.

The run worked mechanically - training completed, artifact landed, contamination clean. Then the eval came back and made us sit in silence for a second:

RU FNR went from 9.11% to 31.72%.

AUROC barely budged (0.99957 → 0.99969). That discrepancy matters. A model whose ranking does not improve cannot be fixed by changing loss weights - you are just moving the cutoff along a fixed curve. Which is exactly what happened: calibrated threshold ended up pushed to 0.99, meaning almost every decision became far more conservative, and the recall collapse followed mechanically.

Threshold sweep showed the real picture. At every grid point where FPR dropped below our 1% cap, RU FNR sat above the 8% gate, and vice versa. The two constraints don't intersect for that configuration. No amount of threshold hunting finds a value that isn't there.

Second attempt: walk away from weights

If pushing loss weight doesn't work, the next reasonable hypothesis is something architectural. Notably, Russian tokenised longer than the other five languages under MiniLM's tokenizer - roughly 30-40% more tokens on average for equivalent semantic content. If sequences were getting clipped at 64 tokens, the tail of an attack might be invisible.

We tested two variants:

• seq_len = 96, everything else default
• seq_len = 128 + lr_head bumped to 2e-3

Both passed. Cleanly. Every gate. In sequence.

That pattern - both sequence-length experiments passing regardless of LR tweaks while all non-seq-length variants kept failing - is exactly what a tokenisation-window story looks like when it shows up in your data.

Four independent configs hit ALL GATES PASS:

Three further candidates failed one or more gates, mostly short-sequence variants where a higher lr was still insufficient.

The right side of that chart is boring by design. Both passing configs are in the same neighbourhood regardless of whether we doubled the learning rate. Sequence length is doing the heavy lifting.

Confirming it wasn't luck

Four passes across a sweep does not prove causation. To check we were looking at a robust recipe rather than a lucky combination of hyperparameters, we retrained the exact winning configuration twice more with different random seeds. Everything else identical.

Calibrated threshold moved (0.50 → 0.91 → 0.80) as expected - probability distributions shift slightly per seed. Gate outcomes did not budge. When different random initialisations produce independent threshold placements but identical hard-gate conclusions, you have a finding rather than an anecdote.

Combined with cross-validation style sweeps above, we now have seven distinct configs passing the full pre-registered suite, all tracing back to the same underlying structural choice.

What the winning run actually looks like

On the frozen regression-holdout set with the calibrated threshold of whichever run we deploy:

• False positive rate: 0.64% measured - 1 in 156 clean negative prompts triggered
• Per-language false negatives: RU 0.56% - IT 0.00% - PT 0.00% - ES 0.00% - DE 0.12% - EN 0.00%
• AUROC over bench-core: 0.99986
• Catch rate on PromptGuard misses: 98.8%
• CPU inference latency (int8 ONNX, batch=1): p50 ≈ 27ms, p95 ≈ 45ms

What this does NOT solve

Being honest about limits is the whole point of publishing these numbers in public.

We tested every model's ability to catch paraphrases of things the rules engine already knows about, plus genuinely novel synthetic attacks generated against this corpus family. Real adversary behaviour shifts faster than any offline dataset tracks. Something will eventually bypass both the regexes and the classifier in ways our holdouts never anticipated. The point of measuring rigorously is to know what you're working with, not to pretend you've solved detection permanently.

Second: the winning configuration uses 96-token sequences instead of 64. CPU inference cost rises modestly (about 45% slower than seq64 on our hardware). There is also no reason sequence lengths larger than this would keep helping indefinitely; the failure mode being addressed is specifically truncation of longer Russian phrasing, and past that window you are paying compute for marginal benefit.

Third: canary and bench-private-holdout partition exclusions held by ID assertion. That contamination discipline survives audits but is fragile against novel corpus sources we haven't yet ingested.

Fourth: nine configurations in this campaign failed one or more gates before any succeeded. Publishing pass-only stories teaches nothing about how to build the pipeline that catches them. Every single fail - including catastrophic ones like the x5-weight collapse to 31.72% - is preserved in the commit log exactly as it ran.

What's next

Shadow-deploying alongside the deterministic rules engine on POST /v30/analyze in OR-combined mode: regex hits or model score beyond threshold yields block, either alone does not necessarily yield allow. Rules engine handles whatever the head misses; neural net catches everything else that regex cannot see. Latency stacking is intentional: regex path costs nothing additional and doubles as a fallback if ONNX runtime misbehaves.

Want to try it yourself? Head to the developer docs — the full /v30/analyze contract, auth setup, and request schema are documented there, along with /openapi.json once shadow metrics stabilize. Client integration notes land in the changelog after staging burn-in.