GitHub Copilot for Kubernetes

This lesson has one measurement at its centre, and everything else follows from it.

A first-draft Kubernetes manifest — a Deployment and a Service for a small API, the shape you get from “give me a Deployment and a Service for this” — was run through two tools:

kubeconform -strict    2 resources found, Valid: 2, Invalid: 0, Errors: 0
checkov                69 passed, 21 failed

The manifest used a :latest image tag, set no resource requests or limits, defined no probes, specified no securityContext, ran in the default namespace, and exposed the service with a LoadBalancer — a public, billable cloud resource.

Strict schema validation reported it as completely valid. That is not a defect in kubeconform. Schema validation answers “does this document match the Kubernetes API’s shape”. Safety is a different question, it needs a different tool, and if you only run the first one your CI is green on a manifest that should not ship.

Where Copilot helps

Manifests are YAML with a published schema and an enormous public corpus, so the structural output is good. Copilot reliably produces a syntactically valid Deployment, gets label selectors consistent, and converts between related resources — a Deployment into a StatefulSet, a Compose service into a Deployment plus a Service — with the mechanical parts correct.

It is also unusually good at the things around manifests: explaining a chart you inherited, interpreting an events list, and turning a CrashLoopBackOff plus kubectl describe output into a hypothesis. Kubernetes produces a great deal of precise diagnostic text, and reading it quickly is exactly where a model earns its place.

Where it consistently needs correcting is everything that is a policy decision rather than a structural one — and Kubernetes has an unusual number of those expressed as optional fields with permissive defaults.

The fields that are omitted by default

Four blocks are missing from essentially every generated Deployment. Each has a default, and each default is wrong for production.

Resource requests and limits. With no requests, the scheduler has no information and will place the Pod anywhere; with no memory limit, a leak takes the node down rather than the Pod. Requests are what scheduling uses; the memory limit is what makes exceeding it fatal to one Pod instead of to its neighbours.

Probes. No readinessProbe means the Service sends traffic to a Pod the moment its container starts, before the application can answer. No livenessProbe means a wedged process is never restarted. No startupProbe means a slow-booting application gets killed by its own liveness probe before it finishes starting.

securityContext. The default is a container running as whatever user the image specifies — root, unless the Dockerfile said otherwise — with a writable root filesystem, all default capabilities, and the ability to escalate privileges.

A namespace. Omitting it means default, which has no Pod Security Admission labels and typically no NetworkPolicy.

Probes: three of them, answering different questions

This is the area where a plausible-looking manifest most often causes an outage, because the failure is intermittent and looks like the application’s fault.

Readiness answers should this Pod receive traffic right now. Failing it removes the Pod from the Service endpoints. Nothing is restarted. This is the probe that should check dependencies.

Liveness answers should this Pod be killed and recreated. Failing it restarts the container. It should check only whether the process itself is wedged.

Startup answers has this finished booting yet. While it is failing, the other two are suspended. It exists so a slow start does not trip liveness.

The classic generated mistake is pointing all three at the same endpoint, where that endpoint checks the database. Now a database blip fails liveness, every replica restarts simultaneously, and a brief dependency problem becomes a full outage — with the restarts themselves adding load to the recovering database.

Practical project: a reviewable Deployment

copilot-kubernetes-demo/
├── manifests/
│   ├── 00-namespace.yaml       Pod Security Admission: restricted
│   ├── 10-configmap.yaml
│   ├── 20-deployment.yaml
│   ├── 30-service.yaml
│   └── 40-networkpolicy.yaml   default-deny ingress and egress
├── manifests-draft.yaml        the first suggestion, kept for comparison
├── README.md
└── .github/
└── instructions/
└── kubernetes.instructions.md

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: copilot-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
    spec:
      # The default ServiceAccount gets a mounted token by default. This
      # workload never calls the Kubernetes API, so it should have neither.
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: ghcr.io/example/copilot-demo-api:0.1.0@sha256:0000…0000
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8000
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            privileged: false
            capabilities:
              drop: [ALL]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              # No CPU limit on purpose: a CPU limit throttles rather than
              # kills, and throttling a latency-sensitive service to enforce a
              # number nobody measured causes more incidents than it prevents.
              memory: 512Mi
          startupProbe:
            httpGet: { path: /healthz, port: http }
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 15
            failureThreshold: 3
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        # readOnlyRootFilesystem means anything that writes needs an explicit
        # writable mount. An emptyDir for /tmp is usually all that is required.
        - name: tmp
          emptyDir: {}

apiVersion: v1
kind: Namespace
metadata:
  name: copilot-demo
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest

                        draft          final
kubeconform -strict     2 valid        5 valid
checkov                 69 passed      90 passed
                        21 FAILED       1 FAILED

Validation

The dry-run assumption that is wrong

It is widely believed that kubectl apply --dry-run=client is a local check. It is not. Running it here, with no cluster configured:

error: error validating "manifests-draft.yaml": error validating data:
failed to download openapi: Get "http://localhost:8080/openapi/v2?timeout=32s":
dial tcp 127.0.0.1:8080: connect: connection refused

kubectl needs the API server’s OpenAPI schema to validate against, so --dry-run=client requires a reachable cluster even though it changes nothing. Adding --validate=false does not help — it then fails resolving API groups instead.

The practical consequence: kubectl --dry-run=client cannot be your CI validation step unless CI has cluster credentials, which is usually a worse trade than the check is worth. kubeconform is the offline validator, it needs no cluster, and it is what the pipeline in the capstone uses.

--dry-run=server is a genuinely useful check — it runs admission controllers and webhooks, so it catches policy rejections a schema cannot — and it needs a cluster by definition.

ConfigMaps, Secrets and how configuration reaches a Pod

Generated manifests get the mechanics right and the lifecycle wrong, and the lifecycle is where the surprises live.

A ConfigMap consumed with envFrom is read once, at container start. Updating the ConfigMap does not change the environment of a running Pod. Nothing warns you; the value in the cluster and the value in the process simply differ until something restarts. Generated documentation frequently claims otherwise.

A ConfigMap mounted as a volume does update, eventually — the kubelet syncs it on its own schedule, typically within a minute or two, and the application has to notice the file changed. This is the mechanism if you want live reload, and it requires the application to cooperate.

The practical answer for most services is to treat configuration as immutable per rollout: change the ConfigMap and trigger a new rollout, so the change is versioned, observable and rollback-able like any other. A checksum annotation on the Pod template is the usual way to make that automatic, and it is worth asking for by name because no suggestion adds it unprompted.

Secrets are base64, and base64 is an encoding. kubectl get secret -o yaml returns the value to anyone with read access to the namespace. Secrets are encrypted at rest only if the cluster is configured for it, which is a cluster-level decision most manifests cannot see.

Three things to check in any generated Secret handling. Is there a real value in data: or stringData: — that is a credential in Git. Is the Secret mounted as a volume rather than injected as an environment variable — environment variables appear in kubectl describe pod, in crash dumps, and are inherited by every child process. And does the ServiceAccount that can read it need to.

Rollouts, and what actually happens during a deploy

A Deployment’s update strategy is a capacity and availability decision that generated manifests leave at its default, and the default is not neutral.

RollingUpdate with the default maxUnavailable: 25% and maxSurge: 25% means Kubernetes may take a quarter of your replicas out of service before new ones are ready. For three replicas that rounds to one — a third of capacity — which is fine if you are provisioned for it and an incident if you are running close to the line at peak.

maxUnavailable: 0 with maxSurge: 1 is the conservative alternative: never fewer than the current replica count, one extra Pod at a time. It needs headroom for one more Pod and it is slower. That is usually the right trade for a user-facing service, and it is what the example above uses.

Three things that interact with a rollout and that suggestions omit:

terminationGracePeriodSeconds and the preStop hook. When a Pod is terminated, the kubelet sends SIGTERM and removes it from Service endpoints — but those happen concurrently, and endpoint propagation across the cluster is not instant. A Pod that exits immediately on SIGTERM drops the requests still being routed to it. A preStop sleep of a few seconds, and an application that finishes in-flight requests before exiting, is what makes a deploy invisible to users. This is also why the exec form of CMD matters in the Docker lesson — the shell form never receives the signal at all.

PodDisruptionBudget. A rollout is a voluntary disruption; so is a node drain. Without a PDB, a cluster upgrade can evict every replica of your service at once because nothing told it not to.

revisionHistoryLimit. Defaults to 10 ReplicaSets retained, which is what makes kubectl rollout undo possible. Suggestions sometimes set it to 0 to keep things tidy, which removes the ability to roll back.

The commands worth knowing, and worth putting in your instructions file as the allowed ones: kubectl rollout status deployment/api to watch, and kubectl rollout undo deployment/api to revert to the previous ReplicaSet. The second is the fastest remediation Kubernetes offers and it only works if the history is there.

Helm and Kustomize, briefly

Most real manifests are not written by hand, and the two mainstream templating approaches change what “review the manifest” means.

Kustomize overlays patches onto a base. It is built into kubectl, the base is plain YAML, and kubectl kustomize overlays/prod renders the result — which means the thing you validate is the rendered output, not the base. That is the key review point: run kubeconform and your policy scan on the rendered manifests, because a patch can remove a securityContext as easily as add one.

Helm templates Go text into YAML, which is more powerful and considerably harder to review, because the artifact is not YAML until it is rendered. The equivalent step is helm template, and the same rule applies — validate the render.

Copilot handles both reasonably. Two failure modes specific to them are worth knowing. In Helm, a value referenced with .Values.something that does not exist renders as an empty string rather than an error, so a typo silently produces image: ":latest" or an omitted field. --strict and a values.schema.json turn that into a failure. In Kustomize, a strategic-merge patch that targets a list — containers, volumes, env — replaces or merges depending on the patch type and the list’s merge key, and getting it wrong drops entries silently.

Kubernetes-specific risks

Missing probes, limits and securityContext. Covered above; the recurring three.

:latest and mutable tags. In Kubernetes this is worse than elsewhere, because a Pod rescheduled onto a new node re-pulls the tag and may get different code from its siblings. Pin by digest.

Liveness checking a dependency. Covered above. Turns a dependency blip into a correlated restart of every replica.

hostPath, hostNetwork, hostPID and privileged. Each punctures the container boundary. A hostPath mount of / is root on the node; privileged: true is equivalent. Generated manifests introduce these when asked for something that “needs access to the host”, and the right response is almost always to ask what specifically it needs.

A LoadBalancer Service. Provisions a cloud load balancer with a public address and a recurring bill. Fine when intended; expensive when it appeared because the example it was drawn from used one.

Wildcard RBAC. verbs: ["*"] on resources: ["*"] in a Role is the same mistake as a wildcard IAM policy, and it appears for the same reason — it makes the tutorial work.

Secrets in manifests. A Secret is base64, not encryption. A generated manifest with a real value in data: is a credential in Git.

maxUnavailable defaults. The default RollingUpdate allows 25% of replicas to be unavailable during a rollout. For three replicas that is one Pod, which may be fine — but it is a capacity decision that was made for you.

Destructive commands

Debugging

This is where Copilot is most useful in Kubernetes, and the reason is that Kubernetes produces exceptionally good diagnostic text — it is just voluminous.

The sequence worth encoding in your instructions file:

1. kubectl get pods — what state, and how many restarts.
2. kubectl describe pod <name> — the events list at the bottom is usually the answer. Scheduling failures, image pull failures, probe failures and admission rejections all appear there.
3. kubectl logs <name> — and --previous for a container that has already restarted, which is the log you actually want in a crash loop.
4. kubectl get events --sort-by=.lastTimestamp — cluster-level context the Pod description does not include.

Pasting the describe output and the logs into chat and asking which of the common failure classes this is — image pull, scheduling, probe, admission, application — is fast and reliable. The categories are distinct and the evidence for each is in different parts of that output.

“Quote the line that supports your answer” is the clause that makes the answer checkable in five seconds rather than plausible in five paragraphs.

Review workflow

Best practices

• Ask for probes, resources and securityContext by name in the prompt; none appears by default.
• Label namespaces with Pod Security Admission so the security fields become admission requirements rather than review items.
• Use kubeconform in CI, not kubectl --dry-run=client.
• Pair readOnlyRootFilesystem: true with an explicit writable mount.
• Give every namespace a default-deny NetworkPolicy.
• Record scanner disagreements inline with a reason rather than suppressing them globally.

Common mistakes

• Treating a green kubeconform run as a passing review.
• Pointing liveness and readiness at the same dependency-checking endpoint.
• Assuming --dry-run=client works offline, then discovering CI needs cluster credentials.
• Leaving a LoadBalancer that appeared because the example had one.
• Committing a Secret with a real base64 value and believing it is encrypted.

Where to go next

GitHub Copilot for Docker covers the image these manifests reference, including why the uid has to be numeric. GitHub Copilot for Terraform covers the cluster underneath, and the capstone wires this validation into a pipeline that runs it on every pull request.