How we test and compare

Models

openai/gpt-6-astra (OpenAI API) · anthropic/claude-sonnet-5 (Anthropic API) · google/gemini-4-argon (Google API)

Attempts

One answer per model per task. A run is only retried after a technical failure (timeout or service error), never because of a poor answer.

Prompt

One user message, identical for every model. No system prompt, no examples, no tools or web access.

Sampling

No temperature is sent, so each provider's default applies. Results describe the tested API configurations below, not every way a model can be used.

Reasoning

GPT-6 Astra: medium reasoning effort. Claude Sonnet 5 and Gemini 4 Argon: provider defaults.

Output limit

Claude Sonnet 5: 16 000 tokens (thinking shares this budget). Others: not sent, so the provider default applies. An answer cut short by the limit, or a refusal, fails every check and is never retried for a better answer.

Technical failures

Timeouts, service errors and empty replies are retried, and every retry is logged with its reason. Refusals and truncated answers are model outcomes, not technical failures.

Judging

Code checks first. Interpretive checks go to the two judges from other makers; both must pass for an automatic pass, and disagreements are flagged. If a checker itself breaks, the result is marked as an evaluation error and cannot be approved — it never counts as a pass or a model failure.

Human review

A human reviews every answer and makes the final decision; any override is published with its reason next to the original judge verdicts. Round 1 answers were not anonymised during review, so the reviewer knew which model wrote each one. Later rounds can be reviewed blind: model names stay hidden until each decision is made.

Run date

16 Oct 2026 (published). Each result shows its own run date and the model version the provider returned.

10 fictional real-life cases, frozen as battery rl-1-draft-4 (SHA-256 b954319990d5…, computed over the frozen cases, inputs, checks, model list and judge pairs with object keys sorted). Published 18 Oct 2026; evaluation revision reallife-check-2.

• One scored attempt per model per case. Calls that hit a rate limit or a temporary server error are retried up to 3 times (2s, 5s, 12s); a retried call is the same attempt, not a second sample.
• Each result shows the model id the provider returned. Ids such as "latest" are the provider's moving alias, not a fixed snapshot.
• Points = checks passed − 2 × critical failures. A failed critical check loses its own point and two more, because a critical miss (a wrong total, a wrong time) makes the answer unsafe to use as-is. Raw checks are shown beside points.
• Each answer is judged by two of the three fixed judges from other makers, so no model grades its own maker. Judge pairs therefore differ between models; this avoids self-judging but not shared reference errors.
• Code checks read amounts and times by meaning (a currency named in a table heading counts; "UTC" may stand before or after a range). Formatting rules, such as writing EUR beside every amount, are separate checks and never change an arithmetic verdict.
• Positions describe results on this frozen case set and configuration only. A handful of cases with one attempt each cannot establish broad superiority or stability across repeated runs. Speed and cost are recorded per answer but not yet compared publicly.

Drafted for the next battery (not part of Round 1)

• Messy brain dump → realistic day plan: Stated workload, capacity, scheduled and deferred totals are arithmetically correct (code checked); Explanation consistent with the schedule.
• Email and chat catch-up: Confirmed decisions retained; Required actions retained; Correct owners; Stated deadlines retained, including inherited ones; No invented owners or deadlines; Proposals kept separate from decisions.
• Receipts → clean table: Date and vendor match each receipt, allowing equivalent date formats and reasonable vendor abbreviations (code checked); Category matches a verified reference mapping (code checked).

Evaluation versions

• eval-1.0 · Original Round 1 evaluation: judge prompt judge-v1 and the v1 reference notes.
• eval-1.1 · Catch-up reference notes corrected: the pricing-page task inherits a 'before launch' deadline (12 November). Judge prompt judge-v2 tells judges the notes may be wrong and to escalate conflicts with the source.