● Free models
Can a free model with the right recipe write code that runs?
Free and local models are good enough for a lot of coding. They slip when they have to wire libraries together on versions they never saw. ProvenPin's bet is that a tested, version-pinned recipe closes much of that gap. We have not proven it yet. This page is how we will find out, in public.
The method: three arms, same tasks
Free model alone
The task, the versions to use and nothing else. This is the baseline: what a free model does on its own.
Free model + ProvenPin
The same model and task, with the ProvenPin MCP server connected so it can search and fetch tested recipes.
Frontier model
A top paid model on the same task, alone. This is the bar the free model is trying to reach.
Hidden tests
Each task is graded by tests the model never sees. Code passes or it does not.
Same tasks, same rules
Every arm gets the same task text, the same time and fix limits, and the same grading.
Tasks with and without a recipe
Some tasks have a matching recipe and some do not, so we also see what happens when ProvenPin has nothing that fits.
Everything published
Task list, model names and versions, every run, and the cost. Good results and bad ones.
Scoreboard
Results comingThe models being tested, with exact names and versions, are listed here with the first results. Nothing shows on this board until its data is published.
Why publish before the results?
Because the result could go either way. In our October token test, recipes did not cut total tokens, and we said so. In an outside test, a top model did just as well without recipes. Writing the method down first means we cannot quietly change the rules after we see the numbers.
Want to try it yourself in the meantime? ProvenPin is free and works with any MCP client, including tools that run local models.