Copilot Workflow Rescue

A worked example

Example result — example answers, not a real plan

Generated tests miss important cases · Python/FastAPI · VS Code

Tests that pass tell you the code ran, not that it is right. The useful question is whether each assertion could ever fail — and the fastest way to answer it is to make the implementation wrong on purpose.

Do these three, in order

1. Break the code on purpose and watch the suite

   Invert a comparison, drop a guard clause, return a constant. Run the tests. Put it back.

   Why first: It is the only cheap experiment that distinguishes a test suite from a coverage number.

   Evidence: Supports weak tests: the suite still passes. Weakens: the suite fails on the exact assertion you expected, and names the behaviour.

2. Name the behaviour before asking for tests

   Write the requirement in one or two lines — valid inputs, boundaries, what happens on failure — and give that to the agent instead of the implementation.

   Why first: Documented first-party guidance is explicit that inferring every expectation from the implementation risks tests that preserve an existing bug.

   Evidence: Supports: you cannot state the requirement without reading the code. Weakens: the requirement already exists and the tests still miss cases.

3. Ask for the cases before the code

   Have the agent propose the case list — valid examples, both boundaries, invalid input, failure paths — without editing files. Review the list, then ask for tests.

   Why first: A missing case is obvious in a list of ten lines and invisible in two hundred lines of generated test code.

   Evidence: Supports: the proposed list is missing a boundary you care about. Weakens: the list is complete and the generated code does not implement it.

Prompt to paste into Copilot

Do not edit any files yet.

I want tests for <function or module> in Python/FastAPI. Its intended behaviour is:
* <rule 1 — valid input and expected result>
* <rule 2 — boundary>
* <rule 3 — what happens on invalid input>

1. Propose a list of test cases from that behaviour: valid examples, both
   sides of each boundary, invalid input, and the failure paths. Do not read
   the expected values off the implementation.
2. Mark any case where the intended behaviour is unclear as a question for me
   rather than choosing an answer.
3. Compare your list with the existing tests and say which cases are missing.

When I approve the list, add the tests using `tests/test_<module>.py`, pytest, existing fixtures (`pytest -q`) — reuse the existing
helpers, keep expected values explicit rather than computing them with the
function under test, and do not change implementation code, add dependencies
or refactor unrelated tests.

Check afterwards

• Each assertion checks a named behaviour, not that execution finished.
• Expected values are written out, not computed by calling the function under test.
• Both sides of every boundary appear (at the limit and just outside it).
• Invalid input and failure paths are covered, not only the happy path.
• Mocks stand in for boundaries — network, clock, disk — not for the thing under test.
• Implementation files are unchanged in the diff.
• A deliberate break in the implementation makes ``tests/test_<module>.py`, pytest, existing fixtures (`pytest -q`)` fail.

Copilot for Python — getting pytest to disagree with the implementation

The testing section of the Python lesson: writing tests that can fail, and the Python habits generated code misses.

The interactive version needs JavaScript. Without it, the four problems it covers are: Copilot seems to ignore my instructions; Suggestions do not follow my project conventions; Generated tests miss important cases; Agent tasks make changes I did not intend — and the checks for each are in the articles and lessons linked below.