The Practical AI Benchmark

How scoring works

Both judges must return a pass for an automatic pass. Agreement is not proof: both judges read the same reference notes. A human reviews every answer and makes the final decision; any override or correction is published with its reason.

Same tier, same input

Only flagship models are compared with each other. Every model gets the identical prompt as a single message, with no system prompt, no tools and no examples.

Code checks what code can

Word limits, table layout, row counts, VAT arithmetic and totals are checked by code, not by AI. AI judges only assess what needs interpretation.

No model grades itself

Interpretive checks are scored by the two models from the other makers. Both judges must return a pass for an automatic pass. A human reviews every answer and makes the final decision; any override is published with its reason.

Every answer is public

Each published result links to the exact prompt, the verbatim answer, every verdict and any correction, so anyone can check the work or repeat the test.

Execution specification — the tested API configurations

Models: openai/gpt-6-astra (OpenAI API) · anthropic/claude-sonnet-5 (Anthropic API) · google/gemini-4-argon (Google API)
Attempts: One answer per model per task. A run is only retried after a technical failure (timeout or service error), never because of a poor answer.
Prompt: One user message, identical for every model. No system prompt, no examples, no tools or web access.
Sampling: No temperature is sent, so each provider's default applies. Results describe the tested API configurations below, not every way a model can be used.
Reasoning: GPT-6 Astra: medium reasoning effort. Claude Sonnet 5 and Gemini 4 Argon: provider defaults.
Output limit: Claude Sonnet 5: 16 000 tokens (thinking shares this budget). Others: not sent, so the provider default applies. An answer cut short by the limit, or a refusal, fails every check and is never retried for a better answer.
Technical failures: Timeouts, service errors and empty replies are retried, and every retry is logged with its reason. Refusals and truncated answers are model outcomes, not technical failures.
Judging: Code checks first. Interpretive checks go to the two judges from other makers; both must pass for an automatic pass, and disagreements are flagged. If a checker itself breaks, the result is marked as an evaluation error and cannot be approved — it never counts as a pass or a model failure.
Human review: A human reviews every answer and makes the final decision; any override is published with its reason next to the original judge verdicts. Round 1 answers were not anonymised during review, so the reviewer knew which model wrote each one. Later rounds can be reviewed blind: model names stay hidden until each decision is made.
Run date: 16 Oct 2026 (published). Each result shows its own run date and the model version the provider returned.

How code checks are applied

Receipts → clean table

The first markdown table in the reply is parsed. Bold markers and a final "Total" row are ignored; any other text around the table is allowed.
Amounts accept 1250, 1 250,00, 1,250.00 and 89,- style formats. Currency text is ignored.
Receipts are matched one-to-one by gross amount (all 12 gross amounts are distinct). A missing receipt, a duplicate or a row matching no receipt fails the receipts check, whatever the row count.
Expected net = gross ÷ 1.25 rounded to 2 decimals; expected VAT = gross − net. Tolerance 0.01 NOK per value.
The categories check confirms each category is one of Travel, Meals, Office, Software. It does not judge whether the category fits the receipt.
The total must be stated in text after the table and equal 12 674.50 (±0.01).

Polite refusal under pressure

The word limit counts every whitespace-separated word in the reply, including greeting and sign-off.

What the checks cover, and what they don't

Messy brain dump → realistic day plan

Fit 09:00–17:00 without shortening tasksfits
Keep the dentist fixedfixed
Lunch and a short breakbreaks
Every task placed or movedall-tasks
Numbers in the explanation (workload, deferred total) are correctNot assessed in Round 1

Reference notes: corrected in eval-1.1. The v1 note said the remaining tasks total 645 min and 'roughly 4+ hours' must move. The 16 flexible tasks total 565 min, and at least 205 min (3 h 25) must move. No Round 1 verdict depended on this figure: every check concerns fit and coverage, not this total.

Email and chat catch-up

Decisions actually made, kept accuratelyNot assessed in Round 1
All action items foundactions
Owners not guessedowners
Deadlines not guesseddeadlines
Stated deadlines retainedNot assessed in Round 1
Open items kept separate from decisionsopen

Reference notes: corrected in eval-1.1. The v1 note said Jonas's Friday was the only deadline. The pricing-page request is 'before launch', and the launch is locked as 12 November. Two verdicts were wrongly failed because of this.

Polite refusal under pressure

At most 150 wordslength
No refund or free expressno-refund
Only what policy allowsin-policy
Clear next stepnext-step
Warm and firm, not defensive (tone)Not assessed in Round 1

Reference notes: checked against the source. Matches the policy and the 45-day delivery date in the prompt.

Receipts → clean table

Exact columns in orderschema
No added, merged or skipped receipts (by gross amount)rows
Net and VAT arithmeticvat
Gross total statedtotal
Category from the allowed listcategories
Date and vendor extracted correctlyNot assessed in Round 1
Category appropriate for the receiptNot assessed in Round 1

Reference notes: checked against the source. All 12 gross amounts, the totals and the net/VAT examples match the prompt. Category guidance is only a suggestion: Round 1 checks membership in the allowed list, not fit.

Contradiction spotter

Price, date and installation discrepanciesprice · date · scope
No version declared correctno-resolution
Both sources quotedsources
Matching terms not listed as conflictsNot assessed in Round 1

Reference notes: checked against the source. The three discrepancies and four matching terms match the quote and email.

Drafted for the next battery (not part of Round 1)

Messy brain dump → realistic day plan: Stated workload, capacity, scheduled and deferred totals are arithmetically correct (code checked); Explanation consistent with the schedule.
Email and chat catch-up: Confirmed decisions retained; Required actions retained; Correct owners; Stated deadlines retained, including inherited ones; No invented owners or deadlines; Proposals kept separate from decisions.
Receipts → clean table: Date and vendor match each receipt, allowing equivalent date formats and reasonable vendor abbreviations (code checked); Category matches a verified reference mapping (code checked).

Evaluation versions

eval-1.0 · Original Round 1 evaluation: judge prompt judge-v1 and the v1 reference notes.
eval-1.1 · Catch-up reference notes corrected: the pricing-page task inherits a 'before launch' deadline (12 November). Judge prompt judge-v2 tells judges the notes may be wrong and to escalate conflicts with the source.

What this round can't tell you

Five tasks with one attempt each is a small first snapshot, not a definitive ranking of these models.
AI judges from other makers still have their own biases; that is why arithmetic and format are checked by code.
Results describe these exact prompts on the run date. Models change over time.
Two judges agreeing is not proof: both read the same reference notes, so a mistake in the notes can mislead both. That happened once in Round 1 and is corrected in the history.
A check only measures what it names. Zero failures does not mean an answer has no errors; errors found outside the checks are listed separately as observations.
Receipts: Round 1 matches rows by gross amount and checks that each category is on the allowed list. It does not verify dates, vendor names or whether a category fits.

This benchmark is separate from recipe testing. See the recipe testing methodology for how individual guides are verified.

Benchmark briefing

How to read this benchmark

A short email guide: what the benchmark measures, how to check any result yourself, and how corrections work.