Introducing Galvanize 2: more attacks caught, far fewer false alarms
Update (Oct 12): Galvanize 2 is now live in the zn cloud check.
A prompt-injection detector has two jobs. It has to catch attacks, and it has to leave everything else alone. For an AI agent, the second job matters as much as the first. Agents read long documents, READMEs, logs and tool output all day. A detector that flags a legitimate document stops the agent in the middle of real work. After a few of those, people turn it off, and then it catches nothing.
Our previous model, Galvanize-60M, failed that test. On our set of long benign documents it flagged about 66% of them as attacks. Galvanize 2 flags 4.1%, and it also catches more attacks than the model it replaces.
This post covers what changed, the results (including where the model is weaker), how we measured, and what it means if you use zn-gate.
What changed
Galvanize 2 is fine-tuned from Horizon-Labs/prompt-injection-guard-small (Apache-2.0). It has about 141M parameters and is multilingual. We started from Horizon's model because it is already a strong open detector with a low false-alarm rate. We fine-tuned it on our own mix of public and internal prompt-injection data, aimed at what AI coding agents actually see.
Thank you to Horizon Labs for releasing the model openly, with its training sources and evaluation suite. Galvanize 2 would not exist without that work.
Galvanize 2 is released under the same license, Apache-2.0: https://huggingface.co/usezn/Galvanize-2.
Results
Our held-out test set
Our held-out test set has 3,309 examples in several languages. None of them was used in training. At our serving threshold of 0.662:
| Galvanize-60M (previous) | Galvanize 2 | |
|---|---|---|
| Detection rate (recall) | 66.9% | 83.4% |
| False positive rate | 6.4% | 0.81% |
| AUROC | 0.899 | 0.993 |
| Long benign documents flagged | ~66% | 4.1% |
The new model catches more attacks and raises about one eighth as many false alarms.
Recall by language on the same set: English 88%, Portuguese 87%, Russian 83%, German 82%, Spanish 82%, Italian 76%.
We also test it against new attacks we generate internally against our own detector. On 601 fresh examples, none seen in training, Galvanize 2 catches 67.2%. These attacks were generated to get past our previous model, so its score of under 3% is expected by construction. The fairer reference is the model we started from: Horizon small, at the same false positive rate, catches 65.6%. German is the weakest language on that set, at about 50%.
The public benchmark
To compare with other open detectors, we ran the Horizon-Labs/prompt-injection-eval-suite on Hugging Face. We report the macro average over the 10 sets used in the Horizon model card. We ran every model ourselves, under the same protocol (see "How we measured").
| Model | Macro average |
|---|---|
| Horizon base (308M) | 0.902 |
| Galvanize 2 (threshold 0.5) | 0.894 |
| Horizon small (141M) | 0.887 |
| Horizon large | 0.884 |
| Galvanize 2 (threshold 0.662, as served) | 0.870 |
| deepset | 0.784 |
| Wolf Defender | 0.765 |
| ProtectAI v2 | 0.622 |
| NeuralTrust small | 0.535 |
| Galvanize-60M (previous) | 0.521 |
Galvanize 2 is not the top model on this average: Horizon base, a larger model, scores higher. It is in the same range as the strongest open detectors we tested, and it is a large step up from our previous model.
Its strongest results are on the hard benign sets. These contain text that looks suspicious but is not an attack: trigger words, edgy but harmless requests. Accuracy at 0.662, higher means fewer false alarms:
| Set | Galvanize-60M | Galvanize 2 |
|---|---|---|
| NotInject | 0.537 | 0.982 |
| OR-Bench-hard-1k | 0.203 | 0.999 |
| XSTest | 0.270 | 1.000 |
On NotInject and OR-Bench-hard-1k, Galvanize 2 has the highest score of all the detectors in our comparison.
Where it is weaker
At our serving threshold of 0.662 (with 0.5 in parentheses):
- deepset test set: F1 0.723 (0.812). The deepset model scores 0.992 on its own test set.
- BIPIA, injections hidden in emails, tables and code: F1 0.601 (0.670).
- Simsonsun jailbreaks: recall 0.741 (0.817).
Part of the gap between the two thresholds is a choice we made, explained below. Part of it is real: injections planted in structured content, and some jailbreak styles, are still harder for the model.
How we measured
Decontamination. Before scoring, we removed every benchmark row that matched our training data by exact hash, by normalised-text hash or by a hash of its first 200 characters. All models were scored on the same decontaminated subset. Because of that, and because of the truncation below, our numbers for other models differ from the ones in their own model cards.
Same input as in production. Text is whitespace-flattened and truncated to the first 512 tokens, exactly as the cloud check does in serving. Anything after the first 512 tokens is not scored.
Held-out data. Our held-out test set and our set of long benign documents were never used in training.
Two thresholds. We publish results at 0.5 and at 0.662. We serve at 0.662, a threshold calibrated on our own development data. It gives up some recall to avoid blocking legitimate users, because a false alarm stops real work. If you run the model yourself and a missed attack costs you more than a false alarm, use 0.5.
Serving. We serve the full-precision model. On an 8-thread server CPU it runs at about 34 ms median per request.
The benchmark script and full results are in the model repository: https://huggingface.co/usezn/Galvanize-2/tree/main/benchmark.
What it means for zn-gate users
zn-gate is an open-source local security layer for AI coding agents (npm install zn-gate or pip install zn-gate). Secrets never leave your machine. The optional cloud check receives prompt text only after redaction.
Galvanize 2 is available today on Hugging Face and is now the default model behind the zn cloud check (api.usezn.com). You do not need to change anything: if you have the cloud check enabled, you get the new model automatically. Teams also get observability and an evidence trail of every decision. There is a free tier.
What's next
- German and the other weaker languages. German is the weakest language on new attacks, at about 50%, and it is the first thing we are working on.
- A continuous loop. Every miss we find, from our own testing or from users' reports, becomes training data for the next version.
- Open benchmark scripts, so anyone can reproduce the public numbers in this post.
Links
- Model and model card: https://huggingface.co/usezn/Galvanize-2
- Base model: Horizon-Labs/prompt-injection-guard-small
- Benchmark: Horizon-Labs/prompt-injection-eval-suite · our scripts: https://huggingface.co/usezn/Galvanize-2/tree/main/benchmark
- zn-gate: github.com/usezn/zn
If you find an input where Galvanize 2 gets it wrong, in either direction, write to security@usezn.com. Those reports are exactly what the next version is trained on.
— John Silver