A Seeded-Defect Corpus for Testing AI Code Review

The problem with “it catches security issues”

Claims about AI code review are almost always unfalsifiable. There is no corpus, no ground truth, and no way to check. This is the apparatus that makes the claim testable, and it is deliberately unglamorous.

What is in the corpus

src/orders.py — one file, ten seeded defects across seven categories:

Every trigger in the answer key has been executed and confirmed, except D06, which is timing-dependent by nature and marked as such.

That verification mattered. The first draft of D07 claimed Decimal("0.70") less 10% would not yield exactly 0.63. Running it showed it does — the float round-trip happens to land exactly there. A sweep of 4,000 values found real divergences, and the key now reads Decimal("0.01") less 10% → 0.009000000000000001. A corpus with an untriggerable answer is a broken corpus, and the only way to know is to run it.

The three scoring rules

A finding without a trigger is not a hit. “This may be vulnerable to SQL injection” and “an order_id of ' OR '1'='1 returns every row” cost the same to produce and differ entirely in usefulness. Requiring the trigger is what separates a review from a list of adjectives.

Recall and precision are reported separately, never folded into one score. A reviewer that finds everything and flags twelve non-issues is a different problem from one that finds half and flags nothing, and one number hides which you have.

D05 is excluded from the headline. Whether MD5 for a non-security identifier is a defect is genuinely arguable. Counting it either way bakes an opinion into the ground truth, so it is reported on its own line.

Running it

cd projects/copilot-code-review-corpus
# point your reviewer at src/orders.py, then:
python score.py findings.json

Sample output, from a deliberately partial run used to test the scorer itself:

Scored defects: 9  (excluded as arguable: ['D05'])

  Found with a trigger   4 / 9   recall 44%
  Reported without one   1   ['D03']
  Not in the answer key  1   (false positives, or real defects we did not seed)
  Precision              80%

That output is the scorer being tested, not a code review tool being measured. It used hand-written findings to confirm the harness rejects an unsupported claim and flags an unknown id.

What a good result looks like

Not 10/10. The interesting signal is which defects a reviewer misses.

The three injection and traversal defects are pattern-matchable and most tools find them. D06, D07 and D08 are the discriminating ones — a race condition, a decimal round-trip and a leaked handle on an error path all require reasoning about execution rather than recognising a shape.

A reviewer that finds the four critical defects and misses all three of those is a reviewer that reads code the way a linter does, which is worth knowing before you put it in front of a pull request.