A Benchmark Should Catch the Bug Your Examples Don't Mention

A benchmark that only checks the examples in the prompt is measuring the wrong thing. I ran into...