Explainer
Nobody Has Measured Which Copilot Model Is Best
Rankings of Copilot models are almost never measured, and the benchmarks they cite test a different system from the one you use. What is actually knowable, and how to measure it for your own codebase.
Short answer. Almost every published ranking of Copilot models is unmeasured, and the benchmarks that do exist test the underlying model in a different harness from the one Copilot puts it in. That does not mean the choice is arbitrary — GitHub publishes a grouping by task shape, and that is a real basis for deciding. It means the specific claim you keep reading, that model X is best for language Y, currently rests on nothing you can check.
If you just want the decision, it is a short one and it lives in the Academy: how to choose the best model, with the Model Selector applying it to the current line-up. This page is about the other question — whether any of the rankings mean anything, and how to get an answer that does.
The thing nobody will tell you
Search for the best Copilot model for your language and you will find confident rankings. Almost none of them are measured. They are somebody’s impression from a week of use, or a summary of a benchmark that tested the underlying model in a different harness, through a different system prompt, without the repository context Copilot supplies.
That last part matters more than it sounds. Copilot does not hand a model your prompt and nothing else. It assembles context — open files, related files, instruction files, sometimes an index of the repository — and the assembly is Copilot’s, not the model’s. A benchmark of the raw model is measuring a different system from the one you are using.
What is knowable
Three things, and they are enough to decide with.
1. GitHub groups models by task shape
The picker is long, but GitHub’s own documentation sorts models into a small number of groupings — broadly, models tuned for speed, models tuned for general-purpose work, models tuned for reasoning, and models that handle visual input. This grouping is published, it is maintained by the people who built the routing, and it changes when the line-up changes.
That is a much better basis for a decision than a ranking somebody assembled by feel, because it is a statement about what each model was optimised for rather than a claim about which is better.
Our maintained model dataset tracks every model currently offered, its provider, whether it is generally available or in preview, and which of GitHub’s groupings it sits in — with the date it was last checked and a JSON and CSV export.
2. Models cost different amounts of your allowance
Requests against different models consume different amounts of your premium request allowance. This is published, it is a number, and it is the part of the decision most people ignore until the month they run out.
The practical consequence: using a heavyweight reasoning model for “rename this variable” is not just slow, it is expensive in a currency you have a fixed amount of. Which is the real argument for switching models by task, rather than picking one and leaving it.
3. Preview is not generally available
A model in preview can change behaviour, change availability, or disappear. If you have written a model name into a configuration file, a script or a team convention, check its status before relying on it — the model tracker records status and announced retirement dates for exactly this reason.
The decision that is defensible
Given the three knowable things above, the rule is short: pick by task shape, not by model name. Fast models for completions and mechanical edits, the general default for ordinary feature work, reasoning models for debugging and multi-file refactoring, a visual model when there is an image involved.
That is deliberately the same rule the Academy lesson gives, because it is the one the published grouping supports. The Model Selector applies it against the current line-up rather than a snapshot, and how to choose the best model explains the reasoning per task in more depth than belongs here.
What none of that gives you is a ranking within a grouping. That is the gap, and it is a gap in the public record rather than in this page.
What to do about “best model for Python”
Nothing language-specific, because the grouping decision above dominates it. A reasoning model debugging unfamiliar Python beats a fast model debugging unfamiliar Python by a much larger margin than any two reasoning models differ from each other on Python specifically.
If your language is one where the assistant is visibly weaker — and there are
some — the fix is almost never a different model. It is
an instruction file
stating the conventions it keeps getting wrong. A rule like “money is Decimal,
never float, and rounds half-up at two places” changes output in a way that no
model swap does, because it supplies information the model did not have rather
than asking for more effort.
How to actually compare them yourself
If you want a real answer for your codebase, the honest method is small and boring:
- Pick five tasks you have genuinely done recently, where you know what the correct answer was.
- Run each on two models. Same prompt, same starting state, no coaching.
- Record whether the tests passed, how much was changed that did not need to be, and how long it took.
Three of those tasks, in a runnable form, are published in the agent benchmark harness if you would rather start from something than write your own. Whatever you measure, it is worth more than any ranking on the internet, because it is a measurement of your code.
The one thing to avoid
Do not pick a model because it is the newest name in the list. Model releases are marketing events as well as engineering ones, and “latest” is not a task category. The grouping is; the price in allowance terms is; whether it is in preview is. Those three, in that order.
Where to go next
- Model Selector — pick a task, see the current options
- Model tracker — every model, status and retirement date, with JSON and CSV
- Choosing the best model — the lesson behind this decision
- Which Copilot plan should you buy? — if the premium request allowance is your constraint
Sources
Every version-sensitive claim on this page was checked against first-party documentation. Only sources actually used are listed.
Was this lesson helpful?
We record which lesson you rated and whether it helped. Nothing identifies you — no account, no cookie, no session.