Before trusting a novelty judge, count the verdicts that move

A judge can keep the same accuracy while changing its mind about half the examples. The average looks...