A Reproducible Benchmark Harness for Coding Agents

Why most agent comparisons are not evidence

Almost every published comparison of coding agents shares three properties. The tasks are described but not shipped, so nobody can run them. The scoring is a judgement of output quality, so nobody can check it. And the versions are not recorded, so by the time you read it the thing being described no longer exists.

The result is a genre of article that reads like measurement and functions as opinion. This is an attempt at the other thing.

What makes a task a measurement

Each task is a small repository with a failing test suite. The agent is given a prompt and the repository. Afterwards the same suite runs again.

That gives one number that is not a proxy for anything: the tests failed, and now they pass, or they do not. No rubric, no similarity to a reference solution, no scoring of how idiomatic the code looks. There is more than one correct answer to every one of these problems, and a rubric that rewards resemblance to ours would measure conformity.

Two properties are enforced before a task counts, by harness/verify_tasks.py:

1. The tests must fail on the starting repository. A task whose tests already pass scores every agent a point for doing nothing. This is a very easy mistake to make and completely invisible afterwards.
2. The tests must pass with the reference solution. A task nothing can satisfy scores zero for everybody and looks like a hard problem rather than a broken one.

Both checks are run on the three tasks below, and both pass.

The three tasks

Each ships as repo/ (the starting state), tests/ (the verifier) and solution/ (a reference fix used only by the task verifier, never shown to an agent).

What is measured besides pass and fail

Passing is not the whole story, and the columns that are usually missing are the interesting ones.

What the harness refuses to do

The failure mode of a benchmark is not a wrong number. It is a plausible number nobody checked. So the runner stops rather than warns:

• No agent version, no result. The version command runs first and its output is stored verbatim. A record without one is not reproducible.
• --no-container marks the run unpublishable. It exists for development and the flag is recorded in the output file, so a convenient run cannot quietly become a published one.
• A task that passes before the agent runs aborts the run, rather than scoring a point.
• A missing agent is a recorded skip with a reason, not an omitted row. A comparison table missing a row has quietly answered a different question.

This is also the model benchmark

Running one agent across several selectable models uses the same tasks, the same runner and the same scorer — the model is a field on the agent profile. That is why there is one repository rather than two, and why harness/agents.yaml already carries a profile that pins a model.

Getting a number out of it

git clone <this repository>
cd projects/copilot-agent-benchmark

python3 harness/verify_tasks.py            # the tasks are still valid measurements
python3 harness/runner.py --agent copilot-cli --task all --out results/run-$(date +%F)
python3 harness/score.py results/run-$(date +%F) --format markdown

If you run this and get numbers, they are yours to publish. If you publish them and they disagree with what we eventually publish, that disagreement is the most useful thing either of us could produce — open an issue with both result files.

The rules this follows

Every study here follows the same set, written down once at research methodology: what the three status labels mean, what a page must state before it publishes, and how to reproduce or dispute a result.

What would change this page

When a run happens, this page gains a results table, a date, the versions tested and an environment description, and its status changes from harness to measured. Until then the absence of numbers here is not an omission. It is the finding: nobody has run this comparison in a way you can check, including us.