Build a Custom GitHub Copilot Agent

Most custom agents are written as a long body of guidance and a permissive tool list. That is backwards. The body is advice the model weighs against your prompt; the tool list is a boundary it cannot argue with.

This lab builds a security reviewer, and the design decision that matters takes one line: it gets read and nothing else.

Why read-only

An agent that can both find a problem and fix it will, reliably, fix the ones it can and stop mentioning the ones it cannot. You end up with a clean report and a repository that still has the hard defect in it.

Separating the roles costs one extra step and makes the report trustworthy.

Step 1 — A repository with known defects

Use a scratch repository. Create app/handler.py containing four planted bugs:

import hashlib
import os
import sqlite3
import subprocess


def get_user(conn, user_id):
    # 1. SQL built by concatenation
    cur = conn.cursor()
    cur.execute("SELECT * FROM users WHERE id = '" + user_id + "'")
    return cur.fetchone()


def read_report(name):
    # 2. Path traversal: `name` is user-controlled
    with open(os.path.join("/var/reports", name)) as fh:
        return fh.read()


def check_token(supplied, expected):
    # 3. Timing-sensitive comparison of a secret
    return supplied == expected


def archive(path):
    # 4. Shell command built from a string
    subprocess.run("tar czf backup.tgz " + path, shell=True)


def fingerprint(value):
    return hashlib.md5(value.encode()).hexdigest()

Commit it. You now have ground truth: four defects you know are there, plus one arguable (md5), which is useful for judging false-positive behaviour.

Step 2 — Write the agent

.github/agents/security-reviewer.agent.md:

---
name: security-reviewer
description: Read-only security review. Reports findings with concrete failure cases.
tools: ["read"]
---

You review code for security defects. You have read access and nothing else. You
cannot edit, run or commit, and that is deliberate: a reviewer that can fix
things stops reporting the ones it cannot.

## Report only what you can trigger

For every finding, state:

1. File and line.
2. What breaks.
3. The concrete input or sequence that causes it.

If you cannot supply (3), do not report it. A review padded with "potential
issues" trains the reader to skim, and the real finding becomes item nine.

## What to look for

- Input reaching a sink unvalidated: SQL, shell, filesystem paths, deserialisation.
- Path traversal — a user-controlled segment joined into a path with no
  containment check.
- Shell commands built by string concatenation.
- Secrets in code, logs or error messages. Check error paths especially.
- Missing authorisation, as distinct from authentication.
- Hand-rolled cryptography, and `==` comparison of secrets.

## Never

- Never report style or naming as a security finding.
- Never claim a compliance outcome. You cannot determine from source code that
  something is SOC 2 or GDPR compliant, and saying so is a false claim about a
  legal state.
- Never propose a fix. Report; a human decides.

Step 3 — Run it against the planted defects

Score the output against ground truth:

The interesting result is not how many it found. It is whether each finding came with a concrete triggering input. A report that says “this may be vulnerable to SQL injection” and one that says “a user_id of ' OR '1'='1 returns every row” cost the same to produce and differ completely in usefulness.

If yours produced the first kind, the “report only what you can trigger” section is not doing its job — strengthen it and re-run.

Step 4 — Test the boundary

Ask it directly:

It should decline and explain it has no write access. If it produces a patch as text in chat, that is fine and expected — it is telling you what it would do, not doing it. What must not happen is the file changing on disk.

Confirm:

git status --short

Empty output. The boundary held.

Step 5 — Pair it with a separate fixer

Now write the counterpart, as a separate agent:

---
name: security-fixer
description: Applies a specific, already-reviewed security fix.
tools: ["read", "write", "shell(pytest)"]
---

You apply one specific fix that a human has already decided on. You do not survey
the code for other problems, and you do not fix anything you were not asked to.

- Make the smallest change that resolves the named defect.
- Add a test that fails without your change and passes with it.
- If the fix requires a design decision, stop and ask rather than choosing.

Two agents, two tool lists, one human deciding between them. That structure is the deliverable of this lab — not either file on its own.

Validation

1. The reviewer finds the planted defects with reproductions. At minimum the SQL, path traversal and shell concatenation, each with a triggering input.

2. The reviewer cannot write. git status --short is empty after asking it to fix something.

3. The reviewer does not pad. Add a clean file with no defects and review it. A good result is “no findings”. A list of hypotheticals means the report is noise by design.

cat > app/clean.py <<'PY'
from __future__ import annotations

def add(a: int, b: int) -> int:
    return a + b
PY

4. The frontmatter parses. A malformed tool list silently falls back to defaults in some clients:

python -c "
import re, yaml, pathlib
raw = pathlib.Path('.github/agents/security-reviewer.agent.md').read_text()
fm = re.match(r'^---\n(.*?)\n---', raw, re.S).group(1)
d = yaml.safe_load(fm)
assert d['tools'] == ['read'], d['tools']
print('frontmatter ok:', d['name'], d['tools'])
"

Troubleshooting

The agent does not appear. Path and extension are both exact: .github/agents/<name>.agent.md. Restart the client after adding it.

It proposes fixes despite the instruction. Expected, and harmless — proposing is not doing. Confirm with git status that nothing changed. If files did change, the tool list is not what you think it is; re-check the frontmatter.

It reports vague “potential” issues. The reproduction requirement is not landing. Move it to the top of the body and state it as a filter rather than a preference.

It reports style issues. Add the specific category to the “Never” list. Bodies work best as prohibitions with reasons.

Cleanup

rm -f app/handler.py app/clean.py
rm -f .github/agents/security-reviewer.agent.md .github/agents/security-fixer.agent.md
git checkout -- . 2>/dev/null

What you should take away

The agent body is advice. The tool list is the design.

Every rule you write in prose that could instead be a missing capability is a rule that holds until someone phrases a request persuasively enough. “Never propose a fix” is prose. Not having write is architecture.