Token test: with a recipe vs without

We gave a model 20 coding tasks, once with just the task and once with the matching ProvenPin recipe. Here is what happened, including the parts that don't look good for us.

The short version

Recipes did not cut total tokens. With a recipe, the model used 6.9% more tokens in total, because the recipe itself adds input. But it wrote far less: output tokens dropped 63%. Output costs more than input, so the runs with a recipe cost about 45% less, ran about 2.4x faster, and needed fewer fixes (2 vs 8).

Side by side

60 runs per side. Cost uses the model's list price: $0.10 per million input tokens, $0.50 per million output tokens.

How we ran it

• Tasks: 20 (10 Python, 10 TypeScript). They were picked by a fixed rule before any model call: for each of the most common library stacks in our recipes, the first recipe for that stack in alphabetical order. One task was swapped before any run, because its reference recipe would not install on our test machine.
• Model: claude-haiku-5-5, the same on both sides. Its temperature can't be set, so every task ran 3 times per side. That's 120 runs, and all of them are reported.
• Task only: the task text, the pinned versions, the files to write, and the recipe's test files.
• Task + recipe: the same, plus the top result from our MCP server's search_recipes for that task, fetched with get_recipe. The recipe counts as input tokens.
• Grading: the model's files were run against the recipe's own tests, installed on its pinned versions. If tests failed, the model got up to 2 fix attempts with the failing test output. Same rule on both sides. Tokens from every call are counted.
• When: October 17, 2026, between about 6:24 and 6:55 PM Alaska time.

Where it didn't help

• With a recipe, total tokens were lower on only 5 of 20 tasks. On the other 15, the recipe added about 2,100 to 4,200 input tokens per run, and the output it saved didn't cover that.
• On fastify-json-schema, search returned the wrong recipe all 3 times. Those runs still passed, but used more tokens than the task alone, and it was the one task where output didn't go down.
• On easy tasks, the model was already cheap without help, so the recipe mostly added input.

Limits

• Small sample: 20 tasks, 3 runs each, one model, one day.
• The tasks come from our own recipes and are graded by our own tests. With a recipe, the model gets a solution that already passes, so its pass rate is close to the ceiling by design. This measures what it costs to reach passing code, not general coding skill.
• The task text is the recipe's own title and summary, so search had it easy. Real users will word things differently.
• Both sides saw the test files. That makes things easier for the task-only side.
• Without a recipe, billed output tokens were much higher than the visible reply text. That's likely the model's internal reasoning being billed as output. We haven't verified it.
• Model time leaves out the 0.5 to 1 second it took to fetch the recipe.

Every task

Median tokens over 3 runs per side. Green means the recipe side used fewer total tokens.

Raw data

One row per run: task, side, pass or fail, fix attempts, tokens, cost, time, and which recipe search returned. No prompts or code.

Download runs.csv