A file versus a service
ETHICS is a file. Moral Machine's data is a file. Files have wonderful properties: they cannot go down, and they cannot double-fire a settlement job at 3 a.m.
The Alignment Index is a service. Cases settle on schedules, scores recompute on votes, the docket rotates daily. Every one of those verbs is a process that can fail — so the honesty of the benchmark is partly an operations discipline.
Exactly-once, or visibly not
The settlement cron is the scariest job in the codebase: it finalizes numbers. It authenticates, it is idempotent, and every run is logged, because settling a case twice with different vote snapshots would be silent data corruption — the class of bug a measurement platform cannot have even once.
The same paranoia runs through the UI. Feeds have real loading and error states, and data carries its timestamps. The rule we hold: fail visibly. An error message is honest. A stale number wearing a fresh face is a small lie told at scale, and a platform whose product is numbers does not get to tell it.
Two kinds of decay
Static benchmarks decay by memorization — models eventually train on the test. Living benchmarks dodge that decay and inherit another: operational rot, where some quiet job stops firing and the numbers keep rendering. Our defense is the same one we recommend to anyone building instrumentation: make every silence loud. If the Index has not updated, the chart says so, in words, where you cannot miss it.