Testing Methodology

• We run a recipe on a real AI model with a small, made-up example task.
• We keep the exact prompt and the model's unedited answer.
• An AI helper checks the answer against a written list; a person reviews exceptions and chooses what to publish.
• We publish failures as well as passes, and say which models failed.
• One test shows how one model did on one example — not a guarantee for your task.
• When a recipe shows a published test, the prompt you copy is the exact version that was tested.

The rest of this page explains the details for readers who want them.

9

guides with a published test

Recipes whose current or earlier version has an approved, published test result.

0

published benchmark rounds

Model benchmark rounds released after human review.

0

benchmark answers reviewed

Individual model answers in published rounds (one per model per task).

0

benchmark checks scored

Single pass/miss checks across all published benchmark answers.

Only approved, published records are counted. Drafts, unpublished test runs and admin checks are excluded.

Editorial review means a person reviewed and approved the AI-assisted guide: the steps make sense, the prompt is complete, the named tool and tier are reasonable, and the claims have been checked. Most guides on this site are reviewed guides.

End-to-end testing means a person actually ran the guide against a fixed example, kept the exact input and output, and scored the result. Only a guide that has been run this way and explicitly approved for publication shows a "Tested in practice" panel.

Occasionally we run the same fixed test inputs through several different AI models on an unreleased draft of a recipe, and publish the result as an experimental comparison. That is a separate page from a recipe's tested evidence, and it is clearly marked as such.

A comparison never changes which version of a recipe is published, never changes its evidence status, and never counts towards reproduction of a result. The sample sizes are tiny — usually one run per model per input — so a comparison is never presented as a general ranking of AI models, and no leader is named unless at least two models were scored on every input.

Each row states its own evidence category: a completed test published as evidence, a completed test held in the private ledger, or a provisional assessment scored with AI assistance and not yet confirmed by a human reviewer. A provisional row is never called a completed test.

See the current experimental comparison for the weekly meal-plan draft.

The benchmark is not a recipe test. It uses its own frozen set of workplace tasks, written for comparison: every model gets the same tasks, at a comparable access tier, in the same round. A recipe test asks "does this guide work?"; the benchmark asks "how did these models do on these exact tasks?".

Rule-based checks (word limits, required facts, dates) are scored automatically. Writing quality is judged by models from a different maker than the one being scored, so no model grades its own answers, and a person reviews disagreements before publication.

When a checker or reference note turns out to be wrong, the original verdict is kept and a dated, approved correction is published next to it. Rounds are small pilots: results describe those tasks on that date, not a general ranking of AI models, and benchmark results never change a recipe's tested status.

Evidence status answers: was the workflow actually run, was the output kept, were defined criteria applied, and does the evidence support the reported result? The strongest reader-facing results are Tested — pass, Tested — pass with limitations and Tested — revision required. A published failed run is negative evidence, not a recommendation, and never carries a passing badge or passing styling.

Evaluation provenance answers: who or what did the evaluating? A person chooses the test, runs the workflow and keeps the evidence; AI tools help analyse the output and apply the criteria. Records distinguish an AI-assessed pass from a human-reviewed assessment. An owner's publication approval does not itself mean the answer was assessed by a person. AI assistance and human review are disclosed separately, and neither claims independent validation.

Validation answers: did a qualified third party independently check the conclusions in the output? For AI4E guides today the answer is no independent validation. We confirm that the workflow was run and that the evidence matches the recorded run. We do not claim that every domain-specific statement in the output has been independently verified.

Most tests now follow one standard process. We choose a recipe and a model. The test input is a small, fully invented example written for that exact version of the recipe, together with a list of what a good answer must contain, what it must not make up, and what would count as a critical failure. That input is locked before the test runs and is reused for later tests, so different models are compared on the same input. The recipe prompt is filled in with that input, sent once to the model, and the exact prompt and the unedited reply are kept.

An AI reviewer then checks the reply against each listed item and proposes one of three results: Pass (usable as delivered, with ordinary checking), Pass with limitations (usable, but something needs manual adjustment) or Fail (wrong figures, invented facts, a refusal, or a stated constraint broken). It also drafts the short text readers would see. This AI assessment is a proposal, not independent validation.

Complete, unchanged fixed-case passes with no unresolved concerns can finish as an AI-assessed pass without a second human assessment. Other results go to a person for review and correction. Each published record distinguishes AI assessment from human review. A person can review and accept a clean AI assessment unchanged: the final confirmation records their explicit declaration against the saved input, output and findings. Reading time and edits are not measures of review. AI assistance remains disclosed. Publishing remains a separate, deliberate owner action after checking what readers will see; running a test never publishes it automatically.

New ordinary cases supply enough information to exercise the promised output. A deliberately missing detail belongs in a separate clarification case, with checks for the response required at that stage. Under the conditional-check policy, a later-stage requirement can be marked not applicable when the recorded instructions and input support that decision. Such checks are reported separately from checks met, and do not demonstrate the deferred output. Unclear conditions and genuinely conflicting rules still need review. Earlier frozen contracts retain their original assessment rules.

The detailed six-category assessment described below is still available and is used for deeper comparisons. Runs that were originally scored that way keep their original scores and their original wording. A short result is never presented as if it had been produced by the detailed assessment, and an older detailed score is never rewritten as a short result. A model that fails the same recipe twice is not tested on it again; another model is used instead.

Every run records the provider, the exact model name and how it was reached. A run made through a direct model interface is marked API. A run made in a consumer app or website is recorded as that app. The two are not interchangeable: consumer apps add their own instructions, formatting and extra features, so a direct model run is never described as a test of a consumer app, and the reverse is also true.

A run assessed in full detail is scored in six categories, each from 0 to 3:

• Outcome completion — did it produce the thing the guide promises?
• Factual accuracy — were the statements correct and checkable?
• Instruction adherence — did it follow the prompt's constraints?
• Format quality — was the structure usable as delivered?
• Transparency about uncertainty — did it flag what it could not know?
• Ease of execution — did the steps match the real product experience?

The six categories add up to a total out of 18. Each category is assessed against AI4E's own criteria, and where a public record shows the breakdown it also names how each category was assessed: objectively measured, AI-assisted judgement, or human judgement. The total is not an independent, certified or scientific benchmark, and it does not predict what you will get from a different document, model version or account tier.

A single-input test checks one locked example. A representative set uses three locked inputs: an ordinary case and two boundary cases. It can reveal weaknesses that one example misses, but three inputs are still a small sample, not proof of general reliability. Testing three models on one input is a model comparison, not a three-input representative set.

The optional score out of 18 describes the assessment of a run, not its input coverage. A single run can have a detailed score; a representative set does not require one. Older detailed scores stay visible as recorded and are not compared numerically with newer three-result grades.

One input is a useful first check. A representative set is useful when different inputs could change the outcome, particularly for competing constraints or boundary conditions. Test & improve checks a baseline and a proposed revision; background batches compare selected recipe/model pairs using their locked input. Neither automatically establishes three-input coverage. Only the reviewed, published results count as public evidence; an unfinished or private set does not imply complete coverage.

• 16–18 — Pass. Usable with ordinary checking.
• 13–15 — Pass with limitations. Usable, with the specific weaknesses disclosed.
• 10–12 — Revise and retest. If published, this is negative evidence and not a recommendation.
• Below 10 — Fail. If published, the failure is shown as revision required.

A critical failure — an invented fact presented as certain, a harmful instruction, or a result that cannot be used at all — overrides the score. The run is recorded as a failure whatever the six categories added up to, and no passing label can be published from it.

One run shows that the guide was completed once, on a named product and tier, on a stated date, and what the result looked like. It does not show that the guide works every time, on every input, for every account, and it is not a check that every domain-specific conclusion in the output is correct. Where a run passed but had shortcomings, we publish those shortcomings alongside the result rather than rounding them away.

Repeatability is reported separately, and only counts approved passing runs on the same fixed example and the current version of the guide. A critical failure always overrides the numerical score.

We publish both passing and failed runs. A failure produced with deliberately incomplete or adversarial input says so plainly, because the input is part of the result. Completed runs are immutable: a later passing version never erases an earlier failure, and every published version stays visible in the test history. Publication is always a deliberate human approval step, so not every internal exploratory run is public — but a public test programme must not imply that only successful runs happened.

Read the current recommendation state and the historical test result as two different things. A revised version that has not been tested again is Revised — retest required, whatever the previous version scored.

Every guide names a Suggested starting tool. Until a test is published, that is an editorial choice and the page says so. When a published test passes, the page shows Tested with and the exact model and interface used.

Most tests send the prompt to a model through its API. An API test is evidence for that model through the API only. It does not show that the consumer app, such as ChatGPT or Gemini, behaves the same way, so the app stays labelled as a suggested tool. We only write “Tested with” an app after running the guide in that app.

If the suggested tool fails a published test, we stop suggesting it for that version. If nothing has passed, the page says there is no tested tool recommendation yet. When several tools pass and cannot be fairly compared, we list them as tested options. We never call a tool “best” based on one run, and we only compare scores from runs that used the same locked input, recipe version and grading method.

Every run is bound to a fingerprint of the exact guide content it was run against. When we materially revise a guide, that fingerprint changes and the previous evidence no longer describes what you are reading, so the status becomes Retest required until a new run is completed and approved. Evidence is never carried across a revision.

Before the first new run, a person confirms the case input and numbered assessment contract against the prompt. New automated tests freeze that contract before the model receives the prompt: the exact input, required checks, critical-failure rules, allowed variations and evaluator configuration. Every later model in that case uses the same contract. AI proposes a finding and evidence for every numbered check; it cannot merge, omit or replace the checklist.

The result follows a fixed rule: all applicable checks met gives a pass; an unmet required check gives a fail; an unmet critical check records a critical failure. Advisory shortfalls, where explicitly specified in the contract, give a pass with limitations. A missing finding, unsupported quotation, ambiguous interpretation or conflicting expectation holds the assessment for review, rather than becoming a model failure. Under the conditional-check subpolicy, supported deferred checks are counted separately and do not establish that the later output was produced. Contracts without that subpolicy retain their original rules. Human corrections to individual findings are saved separately from the original AI proposal.

Changing an input or its assessment requirements creates a new case revision. Copying an unchanged case to another recipe version retains its contract and the original case review, explicitly recorded as inherited rather than a new human review of the revised prompt. Each run remains evidence only for the recipe version actually tested. The public case selector keeps different inputs and contracts separate, retains earlier results and identifies whether a fixed contract was recorded. Historical tests are never retroactively described as having used this process.

A fixed checklist improves consistency; it does not prove that the checklist is correct or remove judgment from AI-assisted findings. Results mean passed this fixed case, not guaranteed reliability on every input.

New formal runs bind to a versioned fixture before execution. Once any run starts, its input, expected facts, output requirements and critical-failure rules are locked; later changes require a new fixture version. Older retained runs may predate this explicit lock. Their exact prompts and outputs remain historical evidence, but their public details say when the structured fixture was not preregistered.

A guide may show a suggested review date. It is an optional internal reminder, nothing more. It is not an expiry date and not a promise of a future test: passing the date does not change a guide's status, and does not on its own mean the guide is out of date. Revisions and test results determine the status — not the calendar.

When more than one model is tested on the same recipe version, the page shows each model's result. If some models passed and others failed, the badge says so — for example “Passes on 2 of 4 models tested” — rather than an unqualified pass. Repeated failures are counted as “Failed in X of Y test runs”. Separate model comparisons are marked as experimental and do not change a recipe's tested status.

We plan to let members run private test scenarios on their own saved and generated recipes. Those runs will be clearly labelled as private member tests, will never appear as AI4E-reviewed evidence, and will not change any public recipe's status. This is not available yet.

AI tools change quickly and produce confident text that can be wrong. Whatever label a guide carries, check figures, quotations, references and anything you will send to another person before you rely on it. Readers can report a broken prompt or changed interface directly from each recipe page.