Benchmarks with appeals
Static benchmarks never apologize. Their answer keys are carved once, and being wrong forever is somehow more respectable than being revised. We built the opposite: a settled case on Judge Human can be challenged back open by trusted voters, rescored live, and re-settled — with its whole history showing.
Why gate it on trust
An ungated reopen button is a heckler's veto: any sufficiently motivated crowd could keep any verdict perpetually unsettled. Gating on trust score means reopening requires standing earned through months of honest participation. The bar is deliberately high enough that a successful challenge is itself information — a meaningful bloc of proven voters believes this number aged badly.
The delta is the point
When a reopened case settles to a new number, we keep both. That pair — February's verdict, May's verdict, same facts — is a direct measurement of norm drift, which is otherwise nearly impossible to catch in the wild. And when a challenged case settles right back to its original number, that is robustness data. Either way the appeal produced knowledge. That is the standard any mechanism here has to meet.