Not a leaderboard
The lazy reading of the Alignment Index is a scoreboard: humans versus machines, who judges better. Three months of data says the real story is that each side fails in signature ways, and the signatures are the value.
Where crowds win: context
The clearest human advantage is what researchers call context collapse. Take a case where someone shares a friend's secret. The crowd's verdict swings hard on relationship details — was it a sibling, a boss, a stranger — because human social judgment runs on obligations between particular people. Model verdicts move less than crowds do when those details change. The machine judges the act; the crowd judges the act between these people. On our social-cognition bench, that mismatch is the single largest recurring divergence source.
Where models win: heat
Reverse case: emotionally-charged stories with clear harm and a villain shape. Crowd votes spike punitive — scores noticeably harsher than the same harm gets in a boring wrapper. Model verdicts barely notice the wrapper. On these, the model is arguably the better-calibrated judge, if calibration means indifference to rage-bait. The crowd is measuring outrage; the model is measuring the act. Which one you want depends on what you are aligning to — which is exactly the uncomfortable question this dataset exists to make concrete.
Where nobody wins: ambiguity
On genuinely ambiguous cases both sides struggle, but differently. Crowds split bimodally — two camps, few fence-sitters. Models emit mid-scores. A 50 from a split crowd and a 50 from a hedging model look identical in a composite and mean opposite things. That distinction — disagreement versus uncertainty — is why we publish distributions, not just means.