Your Eval Suite Has a Flake Rate: Gating LLM Changes on Paired, Powered Tests

I have watched the same release go wrong three times. Someone tunes a prompt or swaps a model, the eval average moves from 0.81 to 0.83, the PR gets approved,…