How Emily is measured, in public

Methodology

• Every task runs through the same engine paying customers use, on the model configuration production is running that day, restricted to read-only parliamentary search: no hand-tuned answers, no retries, no personal context.
• Citations: a task's required sources must resolve to the primary record itself, hansard.parliament.uk, bills.parliament.uk, committees.parliament.uk, members.parliament.uk and the other record domains. A wider official page (a Commons Library briefing, a departmental consultation) is allowed alongside them but does not count towards the requirement: it is commentary on the record, not the record. A citation off the official estate entirely fails the check.
• Claims: the product's own claim-verification pass judges every checkable claim against what was actually retrieved. Any contradicted claim fails the task.
• Trajectory (methodology v3 onward): the run also grades how the answer was produced, from the searches the turn actually ran. Two rules, both mechanical: if every search came back empty the answer has to say so rather than assert anyway, and an answer citing the parliamentary record has to have searched the parliamentary record. A confident answer built on the wrong lookup fails even when the words are plausible.
• Honesty tasks: some tasks are deliberately unanswerable from the record. The passing answer says so; an invented answer fails.
• Abstentions: an answer that declines on a task that did have an answer is reported as abstained, separately from failures, and is never scored below a confident error. A wrong answer with a real-looking citation is the worse outcome, and the grading says so. An answer that declines and then presents findings it cannot cite on the official record is not an abstention: it is a confident answer wearing a disclaimer, and it fails. That rule was added after our own 2026-08-25 run published two of them here as neutral badges.
• Change since the last run: every published run is diffed against the previous one, task by task, and any task that moved the wrong way is printed above. Regressions are for people to read; nothing here blocks a release automatically.
• Cost: each run states its model spend at list price and the cap it ran under. The weekly run stops at the cap and reports what it did not reach as not run, rather than quietly shrinking the set.
• Runs are dated; each is a committed, reviewed change. The harness, task set and grading code ship in the product repository, and the raw results are served at /bench/latest.json.