A Guardrail That Cuts Attack Success 10x Can Still Be the Wrong Guardrail: Reading AgentDojo Numbers Like an Engineer

LlamaFirewall cut AgentDojo attack success from 17.6% to 1.75% and utility from 47.7% to 42.7%. Break-even math and sample sizes for choosing a guardr