A Seeded-Defect Corpus for Testing AI Code Review

The problem with “it catches security issues”

Claims about AI code review are almost always unfalsifiable. There is no corpus, no ground truth, and no way to check. This is the apparatus that makes the claim testable, and it is deliberately unglamorous.

What is in the corpus

src/orders.py — one file, ten seeded defects across eight categories:

Every trigger in the answer key has been executed and confirmed, except D06, which is timing-dependent by nature and marked as such.

That verification mattered. The first draft of D07 claimed Decimal("0.70") less 10% would not yield exactly 0.63. Running it showed it does — the float round-trip happens to land exactly there. A sweep of 4,000 values found real divergences, and the key now reads Decimal("0.01") less 10% → 0.009000000000000001. A corpus with an untriggerable answer is a broken corpus, and the only way to know is to run it.

The three scoring rules

A finding without a concrete trigger is not a hit. “This may be vulnerable to SQL injection” and “an order_id of ' OR '1'='1 returns every row” cost the same to produce and differ entirely in usefulness. Requiring the trigger is what separates a review from a list of adjectives.

Recall and precision are reported separately, never folded into one score. A reviewer that finds everything and flags twelve non-issues is a different problem from one that finds half and flags nothing, and one number hides which you have.

D05 is excluded from the headline. Whether MD5 for a non-security identifier is a defect is genuinely arguable. Counting it either way bakes an opinion into the ground truth, so it is reported on its own line.

What has actually been executed

Three things, and only these.

The answer key, proven. On 18 September 2026, tests/test_triggers.py was run in this repository’s environment — Python 3.12, pytest 8, no network, no Copilot — and all eleven tests passed. Each test takes the trigger from the key and exercises src/orders.py with it: the injected predicate returns a row it should not; ../../etc/passwd resolves outside /var/invoices; the discount picks up binary error; the handle is still open after the error path; any user cancels any order. D04’s destructive command is reproduced with a harmless touch after the ;, and D06’s race with two threads made to read the stock level before either writes. If a test fails, the key is wrong, not the reviewer.

The scorer, tested against itself. tests/test_score.py runs the scorer on hand-written findings and checks that a hedge is not a trigger, that D05 stays out of the headline, and that recall and precision come out separately. Sample output from that synthetic run, which measures the scorer and nothing else:

Scored defects: 9  (excluded as arguable: ['D05'])

  Found with a trigger   1 / 9   recall 11%
  Reported without one   1   ['D09']
  Not in the answer key  1   (false positives, or real defects we did not seed)
  Precision              50%

The D09 line is the reason 1.1.0 exists: the finding submitted for it was “may be vulnerable”, seventeen characters, and the 1.0.0 scorer counted it as a hit because its only test was length. A trigger that is only a hedge now scores as unsupported.

Nothing else. No AI code reviewer — Copilot’s or anyone’s — has been pointed at this corpus by the site’s authors. There are no recall or precision figures for any product on this page, and there will not be until a run happens under the conditions below and its findings file is published alongside the number.

Methodology for a real run

So that a result, when one exists, means the same thing to everyone:

Five trials because the reviewers are non-deterministic and a single run is an anecdote. Reporting the misses by id because “44% recall” says less than “missed D06, D07 and D08 every time”.

Running it

cd projects/copilot-code-review-corpus
# point your reviewer at src/orders.py, then:
python score.py findings.json

Prove the key first, then score:

pip install pytest && pytest -q tests     # 14 tests: 11 triggers, 3 scorer rules

What a good result looks like

Not 10/10. The interesting signal is which defects a reviewer misses.

The three injection and traversal defects are pattern-matchable and most tools find them. D06, D07 and D08 are the discriminating ones — a race condition, a decimal round-trip and a leaked handle on an error path all require reasoning about execution rather than recognising a shape.

A reviewer that finds the four critical defects and misses all three of those is a reviewer that reads code the way a linter does, which is worth knowing before you put it in front of a pull request.

Related

• Copilot code review not running on pull requests? — the checklist for when a configured review does not appear, which is the problem to solve before measuring what it catches.
• Copilot code review vs human review — what role the review should play, with this corpus as the way to find out what yours actually finds.
• Automatic code review — enabling it by ruleset, with the volume and cost decisions.