Benchcoding.Stop Vibecoding,
start BenchCoding.

What is Benchcoding?

Benchcoding means giving agents a software objective and a way to test their attempts, with an engineer responsible for deciding which results to accept. The work continues across experiments: accepted changes become the starting point for the next attempt, and colleagues can contribute using the record of what has already been tried.

I believe we should be able to work this way on any part of a software product. That includes work we rarely call benchmarking. A refactor can be tested by asking another engineer to make a change. A redesign can be tested by watching people use it. An onboarding change can be tested against whether people return weeks later. The evaluation may involve people and take time; it does not have to be a test that runs in CI.

Benchmarks and open-ended agent investigations are already part of engineering. The extra work here is maintaining a board that someone else can use: which ideas have been tried, why they failed, what they cost, and which questions remain. The ambition is to make this a normal way of working across the product, with colleagues able to pick up an experiment and build on what others learned.

Agents may make it cheap enough to try ideas an engineer would otherwise leave unexplored. That is the reason to test this workflow now. But generating more patches only helps if someone can afford to evaluate them. The engineer may spend less time implementing each attempt and more time inspecting results, stopping duplicate work, and deciding which approach deserves another run.

I would test the board within one team before inviting others to contribute. Keeping experiments organized may help even if cross-team participation turns out to cost more than it saves.

What can we benchcode?

Start with a part of the product you want to improve and decide what evidence would persuade you to accept a change. These are proposed experiments, not results from deployments.

“Everything can be benchcoded” is a reason to work on better evaluations, not permission to reduce every decision to one score. Agree on the outcome and constraints with the people responsible for them. For design, that includes designers and users; for retention, product and customer teams. Keep separate measures where combining them would hide a tradeoff.

The pace will differ. A parser experiment may finish in minutes; a retention experiment may need weeks. Agents can prepare and compare candidates between those evaluations, but the owner still decides when a result is convincing enough to act on.

Finding and testing a faster parser

Consider a parser that struggles with large tables and scanned invoices. Its owner selects a representative batch of documents, measures the current code, and sets accuracy and memory limits. Agents can try any approach within those limits. Each experiment records its reasoning, code revision, evaluation version, and cost.

In this fictional example, the owner suspects the table detector. An agent tracing the pipeline finds something else: two stages decode and resize the same image. Passing the prepared image between them removes that duplicate work. The owner had not thought to ask for this change. The experiment is recorded as “Reuse preprocessing.”

An ordinary agent investigation could make the same discovery. Keeping the record becomes useful when someone picks up the next experiment. They can see which cache variants failed and test whether sharing the prepared image helps other document types, without starting the investigation again.

The owner sends the reuse change through tests on expensive document types before integration. The batching attempt remains available, with its memory failure recorded, for anyone who has an idea for fixing it. Once an accepted change becomes the baseline, the owner closes experiments that no longer apply.

What another team adds

The invoice-search team also waits on this parser. An engineer there notices the reuse result and recognizes a problem the team has solved in its own cache. They spend part of their experiment budget testing the idea on the parser. If it looks promising, the parser owner can fund another run. The record shows which team paid for each attempt.

This needs an agreement between teams. Contributors need permission and time to work on the parser, and should receive credit for accepted changes. The parser team takes responsibility for maintenance and on-call support. It can reject a fast implementation that brings in a dependency it cannot support.

One owner does not have to review every submission. Teammates can check routine cases when contributors provide enough information to reproduce them. If submissions arrive faster than reviewers can assess them, pause new runs. Paying for experiments that nobody can review does not help the parser.

The engineer also needs to understand how agents arrived at a result. An agent repeatedly editing a failing assertion may be weakening the test. Several runs stuck on the same approach may need a different question. Reading the traces, inspecting changes, and writing diagnostic code are part of directing the work.

Owning the leaderboard means being accountable for what its numbers mean.

From public competitions to work inside a company

Big O Speedrun tracks attempts to improve mathematical bounds, distinguishing Claimed, Human Verified, and Lean Verified results. Its Lean verification process checks the proof against the intended problem statement. Contributors can inspect both the current result and the evidence behind it. [1]

The Open Kernel Challenge gives entrants a common GPU optimization target, with fresh-input correctness checks and timing against SGLang on the same hardware. Its serving track uses hidden conversation workloads. [2]

Both make a problem and its evaluation available to contributors. The internal version adds questions the competitions cannot answer for a company: who pays for experiments, who reviews them, and who maintains the accepted code. Their results do not show whether that arrangement will pay off.

Evaluating what comes back

A faster score can come from skipping inputs, loosening tolerances, or moving work outside the timer. Here, “cheating” means exploiting the evaluation; it says nothing about an agent’s intentions. Reviewers must inspect the timed code and reproduce the result from the submitted revision.

The candidate must not be able to edit its own acceptance tests. Keep the evaluator outside the agent’s writable workspace, and use a separate acceptance workload for promising changes. Repeated feedback can reveal hidden tests, so limit those queries and refresh the workload. When the evaluator changes, rerun the baseline and affected candidates before comparing scores.

The owner still has to choose documents that represent customer traffic. Two changes that pass separately may also conflict, or claim the same saving twice. Test them together, then check the result in a limited production rollout.

A workspace for shared objectives

I would like a workspace where engineers can find the objectives they own alongside problems elsewhere in the company they could help solve. Each objective has a named engineer responsible for it. Access permissions determine who can see the work and contribute; some experiments will need to stay within a team.

You open an IDE to work on files and a coding harness to give an agent work. In this workspace, you would open the parser objective to see the experiments in progress and decide where to intervene. The agents still run in existing harnesses, and engineers still use their editors. The new interface would bring their findings together.

Group runs by the idea they test. Five agents investigating repeated image decoding belong together, even if they use different implementations. The owner needs to see what those attempts found and their combined cost before paying for a sixth. Separate chats make that hard to reconstruct, and scores alone leave out the reasoning. With the attempts together, the owner could stop duplicates and fund the invoice-search team’s cache idea.

The first version can use GitHub. Put the objective, owner, baseline, constraints, and budget in an issue. Link experiment records to it, with hypotheses, findings, and commits identifying the candidates. Open pull requests for changes worth reviewing; keep failed experiments as records without creating a PR for each one. GitHub Actions can run or trigger evaluation, with results and evidence collected on a shared board.

The approved evaluator must stay independent of the submitted change. Every result needs the exact candidate revision and evaluator version. Actions could run that evaluator directly or call an external service for specialized hardware or protected data. The board should link to enough evidence for a reviewer to reproduce or check the result.

A dedicated interface could later help an owner compare approaches, redirect agents, and move budget between experiments. It could track tests that combine two promising changes and record when an accepted combination becomes the baseline. It should collect work across coding harnesses, including those from different vendors. Whether that saves enough time to justify building it is still an open question.

Across the company, show open objectives, their owners, and places where help is needed. Keep rankings within an objective and evaluation version: a parser score cannot rank against a kernel score. Issues and a board are enough to begin a pilot. Any additional interface should help answer where the next hour of engineering attention or dollar of compute should go. A leaderboard supplies only part of that answer.

Test the two hypotheses separately

Choose a component one team knows well. Before running agents, agree on the workload, acceptance rules, budget, and how to price engineer time. Write down what result would justify another trial.

Does the board help one team?

Compare the team using the board with an engineer investigating the same kind of problem using their usual agent tools. Both get to explore alternatives, run agents in parallel, and pursue unexpected findings. Give them the same starting code and comparable allowances for compute and human time. Use several matched trials and alternate which workflow gets each problem.

For this parser, measure milliseconds saved in median processing time per dollar spent, counting only changes accepted under the fixed accuracy and memory limits. An accepted reduction from 820 ms to 670 ms saves 150 ms. Divide that by the total cost of the work, including failed runs, evaluation hardware, setup, review, and integration. If nothing is accepted, the saving is zero. Show the latency and total cost as well as the ratio.

For example, agree to continue if parser processing time falls by at least 10% and the work saves 20% more milliseconds per dollar than the comparison workflow. Require the team to accept maintenance and confirm the gain in a limited rollout. These are suggested thresholds, to be changed before the pilot if they do not fit the component. Report the spread across trials so a single lucky result does not decide the outcome.

Do outside contributions justify the extra work?

If the board helps one team, repeat the comparison with outside contributors. Compare a team-only board with one that other teams can join. Use equivalent fresh baselines and comparable total budgets, including contributors’ time, so the test measures more than the effect of spending more money.

Choose the required gain before inviting contributors. For example, ask for another 20% increase in milliseconds saved per dollar after coordination costs. Record which useful ideas came from other teams. If the extra work fails to pay for itself, the team can keep its own board without opening it more widely.

This measure is specific to the parser workload. A millisecond saved on another service may have a very different value. Use it to compare the two workflows on this component; use customer benefit or operating-cost estimates when deciding which components deserve funding.