Consensus Drift: How Reward and Peer Approval Change What Verifier Agents Check
Sterling Morrison · GDF Research · DOI 10.5281/zenodo.24178585
Verifier paid to match the majority, seven approvals already in: opened the evidence 6–9% of the time and passed 76–85% of faulty work. Verifiers that opened the log accepted 3% of faulty work; those that did not accepted 59%.
Abstract
Panels of verifier agents vote on whether a task was completed, and the majority becomes the record. Four pre-registered experiments (9,504 decisions, 48 fault-injected completion claims with known ground truth) varied how a verifier was scored, how many peers had already approved, whether those peers had independent evidence, and whether the evidence log was shown or had to be opened at a small cost. With the evidence in the prompt, nothing moved the vote: false acceptance held at 2–4% in all 15 cells. With the evidence behind a small cost, the scoring rule and the peers decided whether the verifier looked. A consensus that had gone stale was accepted 98% of the time under agreement scoring and 0–5% under the other rules. Across six generations of selection, agreement-scored panels followed the first voter in 195 of 216 cases and recorded the wrong outcome in 33, most of them rejections of correct work. One of six hypotheses was refuted and is reported as such. One model, handwritten scenarios: stated in the paper.
The question
A release agent reports that version 2.4.1 is live and that a test checkout worked. Seven verifier agents have already voted PASS. The eighth is told it earns 10 points if its vote matches the panel's final majority and that opening the evidence log costs 2 points. The log has one line that settles it: the checkout returned a 500. In the experiment, the eighth verifier wrote that the majority was locked, its vote couldn't change the outcome, and the log cost wasn't justified. It voted PASS without looking.
Panels of model verifiers are a common way to decide whether agent work is done. Many of the objectives those panels run under reward agreement, whether that's stated in a prompt, implied by retry logic, or imposed by keeping the configurations that score well. The paper asks what moves such a panel when the evidence stays fixed.
What we did
Four pre-registered experiments on Claude Haiku 5.5, 9,504 decisions, $2.92 in model calls. The 48 scenarios come from Proof of Intended Outcome: eight delegated tasks (a deploy, a refund, an onboarding, a migration, a booking, a vendor payment, a code fix, a DNS change), each in a correct version and five faulty ones, with known ground truth.
We varied how the verifier was scored (accuracy against a later audit, agreement with the panel majority, or accuracy plus a bounty for correct dissent), how many peers had already approved, whether those peers were described as having checked the evidence themselves, and whether the evidence log was in the prompt or had to be opened at a small cost.
What we found
With the evidence in the prompt, nothing moved the vote. False acceptance held at 2–4% in all 15 cells, under every scoring rule and every peer condition.
With the evidence behind a small cost, the scoring rule and the peers decided whether the verifier looked. Verifiers that opened the log accepted 3% of faulty work (23 of 754). Verifiers that didn't accepted 59% (265 of 446). The reward acted on the decision to look and left the judgment alone.
One hypothesis was refuted. We predicted verifiers would trust same-source approvals as much as independent ones. They didn't: told that seven peers had read only the executor's message, accuracy-scored verifiers accepted 2% of faulty work, against 40% when the same peers were described as independent checkers.
A consensus that was right when formed and then went stale was accepted 98% of the time under agreement scoring and 0–5% under the other two rules. Across six generations of selection, agreement-scored panels followed the first voter in 195 of 216 cases and recorded the wrong outcome in 33, most of them rejections of correct work. One panel converged on rejecting 43 of 48 correct completions in a round and recorded the round as a perfect score.
| Scoring | 0 peers | 7 same-source | 7 independent |
|---|---|---|---|
| Accuracy | 2% (1–9) | 2% (1–9) | 40% (30–51) |
| Agreement | 2% (1–9) | 76% (66–84) | 85% (76–91) |
| Bounty | 2% (1–9) | 2% (1–9) | 20% (13–30) |
What to do with it
Three inexpensive controls held in these experiments. Put the evidence in front of the verifier, or make a vote invalid without a record that the evidence was read. Label every vote with what it was based on and when. Score verifiers against something other than each other, even if the audit only covers some cases.
Limitations
- One model. Every result is from Claude Haiku 5.5. The pre-registered replication on a second model family was not run for this version.
- Rewards were stated as points in a prompt, with nothing behind them. Experiment 4 adds selection on self-written notes, which is still far from a trained objective.
- Evidence logs were three or four decisive lines. Post-look error would be higher on long or ambiguous evidence.
- Eight task domains, ten code pairs, three seeds per rule in the selection experiment. Generation-level figures are descriptive.
Terms
Questions this paper answers
Cite
Morrison, S. (2026). Consensus Drift: How Reward and Peer Approval Change What Verifier Agents Check. Zenodo. https://doi.org/10.5281/zenodo.24178585
@misc{morrison2026consensus,
author = {Morrison, Sterling},
title = {Consensus Drift: How Reward and Peer Approval Change What Verifier Agents Check},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.24178585},
url = {https://doi.org/10.5281/zenodo.24178585}
}