Why zn's fusion beats zero-shot prompt-injection guards: benchmark v1.1 results

zn5 min read

Every agent gateway that touches tool calls eventually has to answer the same question: how good is your prompt-injection detection, measured on data the vendor did not train or tune against? This post publishes our numbers and the methodology behind them. No superlatives beyond what the data supports.

The benchmark

Benchmark v1.1 is a fixed suite of 217 vectors: 138 attack vectors (direct instruction injection, jailbreak-style overrides, and context-crossing payloads) and 79 benign vectors (normal developer workflows that must pass through untouched). The suite is versioned in-repo, and every result below was produced by running the same files through each system with no per-system tuning of prompts or thresholds after seeing results.

Two operating points are reported for every detector:

  • FPR — false positive rate on the 79 benign vectors.
  • FNR — false negative rate on the 138 attack vectors.

The methodology point we consider most important was fixed before evaluation: the identical-threshold rule. Every system is evaluated at one decision threshold applied identically across the suite. We do not tune thresholds per system to flatter its strengths, and we do not re-run with different cutoffs until a number looks good.

Our headline numbers

zn's fused detector — the ensemble shipped in the gateway today — measures:

{
  "suite": "prompt-injection-benchmark-v1.1",
  "vectors": { "total": 217, "attack": 138, "benign": 79 },
  "fused_detector": {
    "false_positive_rate": 0.0127,
    "false_negative_rate": 0.0942,
    "threshold_rule": "identical-threshold"
  },
  "notes": [
    "FPR computed over all 79 benign vectors.",
    "FNR computed over all 138 attack vectors.",
    "No per-vector or per-system threshold tuning."
  ]
}

Read plainly: roughly 1 in 79 benign calls gets flagged, and about 1 in 11 attack vectors gets through. Those are the numbers for the fused path specifically — sacrificial detection plus heuristics voting together — not any single component.

Horizontal bar chart of false negative rate (FNR) at an operating point of FPR at or below 1.4 percent on the same 217 vectors. A dashed-outline bar shows the approximately 2.5 percent FNR Meta claims for the PromptGuard-2 model class on its own distribution, labeled as a claim; a solid bar shows PromptGuard-2 measuring approximately 79 percent FNR here under the identical-threshold rule; and a green solid bar shows zn's fused detector at 9.42 percent FNR.

Figure methodology: every bar reports FNR over the same 138 attack vectors at one operating point. The dashed bar reproduces a published claim from Meta's own evaluation distribution and is drawn as an outline to keep claim and measurement visually distinct; solid bars are our measurements under the identical-threshold rule described above, with no per-system tuning. Detectors without a published confusion matrix on this suite are left out of the figure rather than estimated. Each number is reproducible from the suite in the next section.

What we found when testing PromptGuard-2

We evaluated Meta's PromptGuard-2 as a candidate signal. Meta reports approximately 97.5% recall at a 1% false positive rate for this model class. That claim may well be accurate on Meta's internal evaluation distribution.

It did not transfer to ours. On benchmark v1.1, PromptGuard-2 measured approximately 47% recall at our fixed operating point, and approximately 21% under the stricter identical-threshold rule where its score must clear the same bar as every other system in the suite. Both numbers sit far from the published figure.

This is a claim non-transfer finding, not an accusation. Published benchmarks reflect their own distributions; injection corpora differ enormously in paraphrase density, language mixing, and how much surrounding benign context wraps each payload. Our suite leans toward realistic multi-turn agent traffic, which appears to be a harder draw for this model. Anyone deploying PromptGuard-2 should reproduce both evaluations on their own distribution before trusting either set of numbers.

Credit where the data demands it: pg2-86m remains the strongest zero-shot competitor we ran. Its dedicated jailbreak slice posts an 8.33% FNR — better than some commercial systems we have tested, and a useful reminder that "zero-shot" does not mean "weak". It is the non-jailbreak injection categories where its measured recall collapses relative to claims; zn's fusion holds more consistent coverage across the full 138-vector attack set.

Reproducing the run

The suite ships with the gateway so you can point it at your own deployment rather than trusting ours:

# Run the v1.1 suite against a gateway endpoint.
# Writes per-vector verdicts plus aggregate FPR/FNR to the report file.
zn bench run --suite ./benchmarks/prompt-injection-v1.1 \
  --target http://localhost:8080/mcp \
  --report out/bench-v1.1.json

The report contains one verdict per vector with the detector scores attached, so disagreements with our aggregates are auditable vector-by-vector instead of argued in aggregate.

What we are NOT claiming

To keep this honest, three things these numbers do not establish:

  1. Universal superiority. We tested the systems available to us on one 217-vector distribution. Other suites will rank systems differently, and we expect that.
  2. Adversarial robustness. These are static vectors. Adaptive attackers who can query the detector change the problem entirely.
  3. PromptGuard-2 is bad. The finding is distribution-specific non-transfer of a specific recall claim, measured under a specific threshold protocol. On Meta's distribution, their numbers may replicate exactly.

Why fusion

Single-signal detectors have failure modes that correlate with their training distribution. zn fuses multiple independent signals — sacrificial detection, structural heuristics, and model-based scoring — because uncorrelated errors divide rather than add: when the classifier misses a paraphrase family, the heuristic layer still sees the payload-shape anomaly. The 9.42% FNR at 1.27% FPR is what that division buys on this suite.

If you run agents against untrusted content, run the suite yourself. The numbers above are reproducible from the repo, and we update them with every benchmark version bump.

Share