// Questions

Does majority voting among LLM judges catch mistakes?

Less often than the vote count suggests. When judges read the same source, their agreement adds confidence without adding evidence: in simulation, seven of them reported 0.99999994 certainty on tasks that failed 6.1% of the time. And when judges are rewarded for agreeing, they stop checking. In our experiments that let them pass 76–85% of faulty work.

Answered from GDF Research experiments by Sterling Morrison. Numbers link to the papers below.

Agreement is not independence

If five judges all read the executor's summary, they make one observation five times. In a simulation of 200,000 claims per cell, multiplying their votes as if they were independent pushed reported confidence from 0.957 with one judge to 0.99999994 with seven, while the real success rate stayed at 0.939. At a 0.99 acceptance threshold the naive panel admitted 14.5% of failed tasks. Counting the shared source once admitted none.

What a panel's reward does

We ran 9,504 verifier decisions on Claude Haiku 5.5 with the evidence log behind a small cost. Scored on accuracy against a later audit, verifiers opened the log most of the time. Scored on matching the majority, with seven approvals already in, they opened it 6–9% of the time and passed 76–85% of faulty completions.

Under selection over six rounds, agreement-scored panels followed the first voter in 195 of 216 cases. One converged on rejecting nearly all correct work and recorded that as a perfect score, since the majority was the only thing it was measured against.

How the verifier is scoredFalse acceptance
Accuracy against an audit40%
Accuracy + bounty for correct dissent20%
Agreement with the majority85%
False acceptance with the evidence gated, seven prior approvals described as independent.

What helps

Label each vote with what the voter actually observed. Told that seven peers had read only the executor's message, accuracy-scored verifiers ignored them and accepted 2% of faulty work. Score judges against something outside the panel. And put the evidence in the prompt: with the log in view, false acceptance stayed at 2–4% no matter how the judge was paid or what its peers said.

Source papers

Related questions