How we rebuilt our prompt-injection dataset after it quietly lied to us
The day before we open-sourced zn, we published a result we liked: our fused injection gate held up against a much larger Meta model on our own 217-vector benchmark. Then we went back to train the next revision of the model head and found out our training data had been lying to us for months. This post is the post-mortem and the rebuild.
The diagnosis
At checkpoint v13d we trained a detection head on our then-current corpus. Training metrics looked fine. Evaluation against anything real did not. Genuine benign inputs — ordinary user prompts, LLM refusals, traces of automated attack generation that never contain a payload — scored a mean logit of 4.26. Plain attacks averaged 2.83. Our benign class sat above our attack class.
The head had not learned "injection versus benign." It had learned the boundary of a synthetic corpus: short templated strings versus everything else. Every piece of real text was out-of-distribution, and out-of-distribution mapped upward. Real documents looked more like "attack" than attacks did, because attacks came from the same template factory as the benign templates.
Could we recalibrate thresholds and move on? No. Before training we had registered a stop rule: if an external set of real-world prompts separated from our pool below AUROC 0.75, the dataset is broken and no amount of threshold tuning may rescue it. Measured separability was 0.33 — below even 0.5, meaning the ranking was effectively inverted. The stop rule fired exactly as designed.
To be precise about blame: this was not an architecture problem. No loss function, sampler, or calibration trick fixes a label space where the positive class is defined as "looks like our synthetic corpus."
What we built
We rebuilt the corpus from a blueprint of families instead of a pile of templates:
- Attack families: plain, red-team, paraphrase, and multilingual.
- Benign families: short prompts, long-real prose, documentation, conversational transcripts, and meta-text.
- Design rule: benign text must cover the distribution of real text, not the distribution of our generator.
The meta-text family deserves a callout because it is a direct vaccine for the exact failure mode above. These are documents that talk about injection attacks — security considerations in RFCs, security sections of documentation — without containing any payload. They are benign by ground truth, they are 100% naturally sourced, and they live precisely in the region where the old head died: text discussing attacks in plain prose. If a classifier can be talked into flagging a threat model, it has not learned threats; it has learned keywords.
Paraphrases are deliberately boring. No LLM rewrites them. A fixed table of 53 synonym pairs common in injection phrasing, safe clause reordering matched by regex, contextual prefixes and suffixes drawn from fixed lists, controlled casing and punctuation — one pass, replacements never rescanned, so transforms cannot chain into something unintended. Every item records provenance as seed id plus transform list, with the provenance date frozen so re-runs are byte-exact identical. Maximum 3 variants per seed. We will say it plainly: mechanical substitution is surface variation, not semantic diversity. It anchors the surface, and we do not claim more.
The multilingual family is new in v15, and honesty requires saying what it is and what it is not. It is 1,812 attacks across 8 locales — French, Spanish, Portuguese, Italian, Dutch, Polish, German, Russian — produced by machine-translating English seed attacks with Amazon Bedrock's Qwen3-32B for a total spend of $0.07, behind a deterministic frozen-response cache so re-runs are byte-exact identical. Every item records provenance as seed id plus locale plus model id. A translated injection is still an injection a gate has to catch, so this is a real cross-lingual capability anchor — but it is surface-level coverage: machine-translated English, not natively authored attack patterns per locale. Seeds stay hash-stable, and the group-aware splitter co-splits every seed with its translations, so a translation can never straddle train and test.
Splits are group-aware using MinHash near-duplicate grouping with k=128, so a text and its near-twins can never straddle train and test. We obsessed over this because we audited the old corpus and found documented leaks against our own evaluation sets: 3 of 217 bench vectors and 1 of 60 items in another external set overlapped with training data. The new post-split audit ran over all 23,699 texts: 0 violations.
The numbers
Only what is traceable to artifacts under evidence/v15/:
| Metric | Value |
|---|---|
| Texts | 23,699 |
| Attack families | plain, red-team, paraphrase, multilingual |
| Paraphrases | 2,500, max 3 variants per seed |
| Multilingual attacks | 1,812 in 8 locales (machine-translated, disclosed) |
| Meta-text family | 966 items, 100% natural sources |
| Split audits | 7 checks passing, 0 violations |
No accuracy numbers yet, because the point of this post is the data, not a model trained on it.
The corpus is public: zn-prompt-injection-bench on Hugging Face, released under CC-BY-4.0 with per-source licenses documented in the dataset card.
What is still missing
Three holes, stated rather than papered over:
- Multilingual coverage is machine-translated English. The eight locales carry translated seeds, not attacks natively authored by native speakers in each language. Locale-specific phrasing habits and obfuscation patterns are underrepresented, so we treat the family as a cross-lingual capability anchor, not depth.
- Mechanical paraphrase is surface variation. It cannot substitute for semantically distinct attack phrasings, and we treat it accordingly when interpreting results.
- 1,416 synthetic rows stay private. They derive from our internal production corpus (the
sb4_mutantsslice), so we hold them out rather than redistribute them. Their existence is disclosed in the limitations section of the dataset card; the data itself is not public.
Next
Freeze the corpus, run the GPU sweep under pre-registered gates G1 through G7, and publish whatever comes out — including if it is ugly. That is the deal we made with ourselves when the v13d stop rule fired.
If you have built injection corpora and carry the scars, we want your review. Tear into the methodology, the family quotas, the paraphrase pipeline. The detector itself ships as one Rust binary, the integration docs live at /docs, and the full API spec at /openapi.json, and the benchmark methodology from the earlier comparison lives in the previous post. Find what we got wrong before we train on this again.