Outside test: a separate agent checks our work
A separate AI agent session wrote 100 everyday coding tasks, then searched ProvenPin for each one, ran the recipes, and compared a coding agent with and without ProvenPin. Every number on this page comes from the published data, and most link to the exact file.
The short version
The recipes are real, but they did not help a top model, and search has a big gap. Every recipe the tester ran passed its own tests and matched its published fingerprint. When a fitting recipe existed, search almost always found it (in the top three for 44 of 45). But only 45 of the 100 tasks had a fitting recipe, and for the other 55, search never said "no match". In the head to head, a frontier model passed every task with or without recipes.
What the tester found
- Recipes run. All 194 recipes the tester downloaded passed their own tests and matched their published fingerprints (recipe_runs.csv, integrity.csv). 9 first needed a setup fix, 8 of them because they need a newer npm than the one that ships with Node 22 (report, section 1).
- Search finds a fitting recipe when one exists. It was the first result for 39 of 45 covered tasks and in the top three for 44 (score_out.json, tasks.csv).
- Coverage is narrow. 45 of 100 tasks had a fitting recipe: 28 of 50 in Python, 17 of 50 in TypeScript (coverage_final.jsonl).
- Search never says "no match". For all 55 tasks with no fitting recipe it still returned results. For 34 of them, the first result looked relevant but would lead a developer the wrong way (tasks.csv, column uncovered_outcome; score_out.json).
With ProvenPin vs without
29 tasks, 3 runs per side, same model and effort on both sides. Both sides passed every run on the first try. With ProvenPin, runs used about 8% more tokens and took longer, but wrote about 11% less code (score_out.json).
| Without | With ProvenPin | |
|---|---|---|
| Passed on the first try | 87/87 | 87/87 |
| Passed in the end | 87/87 | 87/87 |
| Mean total tokens per run | 68,994 | 74,303 |
| Median time per run | 65.1 s | 76.1 s |
| Mean code written per run | 3,618 chars | 3,206 chars |
87 runs per side. One row per run in arms.csv; totals in score_out.json.
How it was run
- Who: a separate AI agent session (
claude-opus-5-5), on October 18, 2026, from 03:50 to 06:05 UTC (run_info.json). - Tasks: 100 (50 Python, 50 TypeScript), written by the tester before it looked at our recipes (tasks.jsonl). The 29 head to head tasks have acceptance tests, frozen before any search (acceptance_manifest.json).
- Head to head: a fresh coding agent per run, with the same task text. The ProvenPin side could search and fetch recipes. Neither side could install packages, run code or use the web. The tester graded the files with its own tests.
- Check it yourself: the tester's own score.py, run on the published files, rebuilds every score byte for byte.
Limits
- One model, and a frontier one. It passed every task without help, so this test could not show whether recipes help. It says nothing yet about free or small models; that is the next test.
- Not fully at arm's length: our side wrote the test brief, set the session's network access and supplied the small MCP client it used. The report also notes a link between the test brief and someone on our team (report, section 7).
- The coding agents could not run their code, which is stricter than a normal session.
- One day, 100 tasks, 29 in the head to head. Names and private paths were removed from the data; the README lists every change.
Raw data
The full folder: tasks, search results as returned, every score, and the tester's report.
Earlier: our own October token test.