Runbooks

One symptom per page. Confirm it, follow the branch the output points to, and stop when the check says the incident is over.

• 502 Bad Gateway nginx: meaning and how to fix it

  nginx 502 Bad Gateway means the upstream sent no valid response. We reproduced 8 causes in Kubernetes; the error.log line tells you which one you hit.

• 504 Gateway Timeout nginx: meaning and how to fix it

  504 Gateway Timeout means nginx gave up waiting on the upstream. Tell a slow upstream, an unreachable one and a too-low timeout apart, with real logs.

• Kubernetes alert fixed, errors continue: find the second fault

  The alerted pod is explained, yet users still get errors: a second, unalerted fault. Scope past the alert with real lab output from two faults at once.

• Kubernetes sidecar OOMKilled: pod 1/2 Ready, app healthy

  A sidecar OOMKilled under its own 16Mi limit left a healthy api 1/2 Ready and its Service empty. Find the container, fix its limit, verify. Lab-tested.

• Kubernetes dnsPolicy None: Name does not resolve, the fix

  New pods log Name does not resolve for a Service that exists. The template set dnsPolicy None with search domains for another cluster domain. Lab-tested.

• Kubernetes Service selects the wrong pods: label overlap fix

  A Service selects pods by label, not by Deployment. Its endpoints list one pod more than the Deployment runs and that pod returns 500. Confirm and fix it.

• Kubernetes memory leak OOMKilled: tell a leak from a small limit

  OOMKilled on a steady interval with clean logs and no spec change is a leak. Tell it from a small limit by cycle length and growth, then fix the code.

• Kubernetes Service connection refused: targetPort mismatch fix

  Endpoints list the pod IPs, yet nginx logs Connection refused and returns 502: the Service targetPort points where nothing listens. Confirm it and fix it.

• Kubernetes ConfigMap change not applied to all pods: the fix

  After a ConfigMap edit, only pods started since the edit run the new config; others keep what they loaded at startup. Compare per-pod logs, fix, restart.

• Kubernetes pod stuck in Init:0/1: why and how to fix it

  A pod stuck in Init:0/1 has an init container that never exits, often a wait loop on a missing Service. Read its logs, check the Service, fix the spec.

• Kubernetes pod Pending, FailedScheduling Insufficient cpu: fix

  A pod Pending with FailedScheduling Insufficient cpu asks for more CPU than any node has. Read the event, compare request and nodes, fix the request.

• Kubernetes PVC Pending: StorageClass not found, how to fix it

  A PVC stays Pending when its storageClassName names a class the cluster lacks. The pod never schedules and whatever depends on it fails. Lab-tested.

• Kubernetes NetworkPolicy blocking traffic: timeouts and the fix

  Requests time out while every pod is Running: a NetworkPolicy lets only other pods reach the dependency. Read the error, find the policy, fix it.

• Kubernetes exceeded quota: ResourceQuota blocks a rollout, fix

  A Deployment rollout that never progresses, with FailedCreate and exceeded quota on the new ReplicaSet. Confirm the quota, give it headroom, verify.

• Kubernetes CreateContainerConfigError: meaning and how to fix it

  CreateContainerConfigError means a Secret or ConfigMap the pod references is missing. Read the event, create the object or roll back, verify. Lab-tested.

• nginx [emerg] unknown directive in Kubernetes: cause and fix

  nginx [emerg] unknown directive means a config line nginx does not know, usually a typo in a mounted ConfigMap. Find the line, fix it, restart, check.

• Kubernetes liveness probe failed, connection refused: the fix

  Liveness probe failed with connection refused while the app serves traffic: the probe points at a port nothing listens on. Confirm, fix the port, verify.

• Kubernetes Name does not resolve: wrong Service name, the fix

  The app exits with Name does not resolve because an env var names a Service that does not exist. Confirm it, rule out CoreDNS, fix the name. Lab-tested.

• Kubernetes Service has no endpoints: meaning and how to fix it

  A Service with no endpoints has a selector that matches no pods, so its ClusterIP refuses connections and the proxy returns 502. Confirm it and fix it.

• Kubernetes OOMKilled (exit code 137): meaning and how to fix it

  OOMKilled means the kernel killed a container at its memory limit. Five causes reproduced on Kubernetes 1.37, plus the exit 137s that are not OOM.

• Kubernetes ImagePullBackOff: what it means and how to fix it

  ImagePullBackOff: the kubelet could not pull the image and retries up to 5 minutes apart. The exact event for 10 causes, captured on Kubernetes 1.37.