Kubernetes CrashLoopBackOff: meaning and how to fix it

CrashLoopBackOff means a container keeps stopping and the kubelet is waiting before it starts it again: the first restart is immediate, then the delay is 10 seconds, doubling each time, capped at 5 minutes. It is not a cause. The cause is in the container's last termination: run kubectl describe pod <pod> and read Last State, its Reason and Exit Code, then the log of the crashed instance. OOMKilled with 137 is the memory limit, Error with 137 plus a Killing event is a failed liveness probe, and Error with 1 is the application exiting on its own, with the reason in its last log lines. Fix that cause; the backoff stops by itself.

At a glance

Every row below comes from a failure we injected on a kind cluster and captured. Ordered by how often it showed up in our runs.

Not CrashLoopBackOff, though often filed under it: ImagePullBackOff, CreateContainerConfigError and Init:0/1. The container never ran, Restart Count stays 0, and there is no log. They are covered under "Is it really CrashLoopBackOff, or ImagePullBackOff?" below.

What does CrashLoopBackOff mean?

It means the kubelet is in the middle of a restart delay for a container that keeps stopping. The pod lifecycle documentation says the state "indicates that the backoff delay mechanism is currently in effect for a given container that is in a crash loop, failing and restarting repeatedly", and that the kubelet restarts exited containers "with an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes)". After "10 minutes without any problems, the kubelet resets the restart backoff timer for that container."

We measured it. A busybox container that prints an error and exits 1, under a Deployment, on Kubernetes v1.37.0 with default kubelet settings (backoff-poll.tsv in the reproduction folder, 2026-10-29, times UTC, start times from the container's startedAt). The kind cluster ran on a Docker VM on a laptop that was shared with several other kind clusters, with the disk nearly full, so read the extra seconds as an upper bound, not as typical kubelet latency:

The first restart is immediate, as the docs say ("Kubernetes attempts an immediate restart based on the Pod restartPolicy"). After that each gap is the documented 10, 20, 40, 80, 160 or 300 seconds plus 1 to 23 seconds, on that shared, loaded VM. The poller sampled once a second and logged only changes, so not every delay message was caught. The current delay is printed in the container's waiting message, which is the quickest way to see where a pod is in the sequence. Our poller (poll.sh) read the whole .status.containerStatuses[0].state object; at 16:00:07 it returned:

{"waiting":{"message":"back-off 5m0s restarting failed container=app pod=exit1-7cd4f4c959-4j5lg_crash(49038db1-a56f-4a90-bca4-add834f142d0)","reason":"CrashLoopBackOff"}}

To print only the message on your own pod: kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].state.waiting.message}'.

Two practical consequences. A pod about six restarts in is retried only every five minutes, so after you fix the cause a stuck pod can take up to five minutes to pick it up; deleting that one pod (its controller recreates it) starts it immediately. And to call a fix done, the container has to run past the 10 minute mark, because a crash at minute 9 continues the old sequence.

The 10 s and 300 s numbers are defaults, not constants. The same page documents a per node kubelet setting, crashLoopBackOff.maxContainerRestartPeriod (beta and on by default since v1.35, any value from 1s to 300s), and an alpha gate ReduceDefaultCrashLoopBackOffDecay (since v1.33, off by default) that changes the defaults to 1 s and 60 s. On a managed cluster you may see shorter gaps than the table above.

The restarts themselves are not the bug. A container exits, the kubelet applies the pod's restartPolicy, and the delay protects the node. The question is always why the container stopped.

Why does kubectl get pods show Error or OOMKilled instead of CrashLoopBackOff?

Because the STATUS column shows the container's current state, and between restarts that state is often still terminated. The pod lifecycle page says only that CrashLoopBackOff "may appear in the Status field of some kubectl commands". In our cluster it appeared for part of each delay, not all of it. Our seven crashing reproduction pods, listed together (triage-columns.txt, 15:56:34 UTC):

NAME                              READY   STATUS              RESTARTS       AGE
exit0-65f8b9dbd-8wdbf             0/1     Completed           5 (2m2s ago)   3m37s
exit1-7cd4f4c959-4j5lg            0/1     Error               5 (119s ago)   3m37s
exit1-fallback-58fc76cdbf-f8dh4   0/1     Error               5 (84s ago)    2m46s
exit127-787659dffb-4n9cj          0/1     Error               5 (113s ago)   3m37s
nobinary-699784c959-rvxvc         0/1     RunContainerError   5 (38s ago)    3m37s
oom-f47d67486-vcc8v               0/1     OOMKilled           5 (2m2s ago)   3m37s
oom-fallback-6ddf9cd97d-nx4rx     0/1     CrashLoopBackOff    2 (29s ago)    43s

All seven are in a crash loop; one says CrashLoopBackOff.

This is not an artifact of one machine. In the lab captures taken before each investigation (50 seconds to 3 minutes after the fault), 7 of the 22 looping pods across the 18 runs read CrashLoopBackOff (05 r1, 09, 10 and 16 in both runs). The other 15 read OOMKilled (8, scenarios 01, 02 and 12, for example 0/1 OOMKilled 3 (34s ago) in 01), Error (1, 05 r2) or Running (6, the liveness case in 06, 1/1 Running 1 (49s ago), and the slow leak in 17). On our cluster today a 06 pod first showed CrashLoopBackOff when it was seven minutes old, on its fifth restart (06-liveness-poll.txt, same loaded VM):

NAME                  READY   STATUS             RESTARTS      AGE
api-c7b987b4d-4v44f   0/1     CrashLoopBackOff   5 (1s ago)    7m11s
api-c7b987b4d-vbv4d   1/1     Running            5 (69s ago)   7m10s

So search by restarts, not by the word. RESTARTS above zero and climbing, or a BackOff event, is the signal:

kubectl -n crash get events --field-selector reason=BackOff

LAST SEEN   TYPE      REASON    OBJECT                                MESSAGE
40s         Warning   BackOff   pod/exit0-65f8b9dbd-8wdbf             Back-off restarting failed container app in pod exit0-65f8b9dbd-8wdbf_crash(42e9f3fd-de24-4e9c-864e-f228dad33a91)
36s         Warning   BackOff   pod/exit1-7cd4f4c959-4j5lg            Back-off restarting failed container app in pod exit1-7cd4f4c959-4j5lg_crash(49038db1-a56f-4a90-bca4-add834f142d0)

(trimmed to two of seven lines)

Is it really CrashLoopBackOff, or ImagePullBackOff?

Check Restart Count. A crash loop needs a container that started. Three failures in the lab look similar in kubectl get pods and are not crash loops:

ImagePullBackOff has a backoff of its own, which is where the shared name comes from: the images documentation says the kubelet keeps retrying the pull "with an increasing back-off delay" up to "300 seconds (5 minutes)". The difference is that CrashLoopBackOff retries a container that ran and stopped; ImagePullBackOff retries a pull that never produced one. A crashing init container is a real crash loop, shown with an Init: prefix. Our reproduction, an init container that exits 1, went from Init:Error to this (delete-pod.txt):

NAME                        READY   STATUS                  RESTARTS      AGE
initcrash-9c5cbd456-92h5x   0/1     Init:CrashLoopBackOff   4 (69s ago)   2m40s

Everything on this page applies to it with -c <init-container-name> (kubectl -n crash logs deploy/initcrash -c migrate printed migrate: cannot connect to db).

One more look-alike: RunContainerError with a climbing restart count is a crash loop. The runtime tried to start the process and failed. It is covered below under exit code 128.

How much is affected, and since when?

List the workload's pods with their ReplicaSet hash. In four of our nine scenarios (05, 09, 10, 16) the crash loop came from a rollout, and the old pods were still serving. Scenario 10, captured before the investigation (get-all.txt, run r1):

pod/api-869d87b484-9n7bc        0/1     CrashLoopBackOff   2 (17s ago)   61s   10.244.1.18   aurora-lab-2-worker    <none>           <none>
pod/api-f64486c4c-hb8mm         1/1     Running            0             69s   10.244.1.15   aurora-lab-2-worker    <none>           <none>
pod/api-f64486c4c-n4zt7         1/1     Running            0             69s   10.244.2.22   aurora-lab-2-worker2   <none>           <none>

One new pod (869d87b484) loops; two old ones (f64486c4c) are fine, because the rolling update stops when the new pod never becomes Ready. User impact is small until something restarts the old pods. Do not delete them to "clear" the alert. The change is whatever differs between the two ReplicaSets:

kubectl -n shop get rs -l app=api
kubectl -n shop rollout history deploy/api --revision=2

On our cluster today, with scenario 06 broken (06-rollout-history.txt, trimmed to the probe lines):

NAME            DESIRED   CURRENT   READY   AGE
api-c7b987b4d   2         2         1       17m
api-f64486c4c   0         0         0       17m

    Liveness:	http-get http://:8081/healthz delay=15s timeout=1s period=10s #success=1 #failure=3
    Readiness:	http-get http://:8080/healthz delay=0s timeout=1s period=5s #success=1 #failure=3

Revision 1 is the lab's base manifest (app/shop.yaml), which has port 8080 on both probes; the difference is the change.

When every pod of the workload loops (01 after kubectl set env, 02 after a sidecar was added, 06 after a probe change), the rollout completed because the new pods started, and the old ReplicaSet is already at 0; the change is still in the template diff. When the looping workload's template did not change in any way that matters, look at what it depends on: in 09 the api's new ReplicaSet came from a plain rollout restart, so the two templates differ only in the restart annotation, and the api crashed because redis had been moved onto a PVC that never bound.

Which cause is it?

Three reads, in this order. Each one is cheap and usually ends the search.

1. Read the last termination. One command for a whole namespace (run here against the reproductions):

kubectl -n crash get pods -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,WAITING:.status.containerStatuses[0].state.waiting.reason,LAST:.status.containerStatuses[0].lastState.terminated.reason,EXIT:.status.containerStatuses[0].lastState.terminated.exitCode'

POD                               RESTARTS   WAITING             LAST         EXIT
exit0-65f8b9dbd-8wdbf             5          <none>              Completed    0
exit1-7cd4f4c959-4j5lg            5          <none>              Error        1
exit1-fallback-58fc76cdbf-f8dh4   5          <none>              Error        1
exit127-787659dffb-4n9cj          5          <none>              Error        127
nobinary-699784c959-rvxvc         5          RunContainerError   StartError   128
oom-f47d67486-vcc8v               5          <none>              OOMKilled    137
oom-fallback-6ddf9cd97d-nx4rx     2          CrashLoopBackOff    OOMKilled    137

[0] is the first container; in a multi-container pod use kubectl describe pod and read every container, because in scenario 02 it was the sidecar, not the app, that was being killed. The exit code alone is not proof of a cause: 137 is 128 plus signal 9 and only says the process was killed. The Reason says by whom. How to tell the sources of a 137 apart is in OOMKilled and exit code 137.

2. Read the log of the crashed instance, with or without --previous. The usual advice is kubectl logs <pod> --previous. That is right while the pod is waiting or running again. While the current container is still terminated (see the STATUS section above), the crashed instance is the current one, and --previous asks for the one before it, which may already be gone. From our reproduction (logs-1.txt):

kubectl -n crash logs pod/oom-f47d67486-vcc8v

loading cache

kubectl -n crash logs pod/oom-f47d67486-vcc8v --previous

unable to retrieve container logs for containerd://46b1ed7c966d4da2e9b49be2f6bc500bbe4b00af1c727ac1a6576ab713a749e2

The same pair returned the same result for the exit 1, exit 127 and exit 0 containers. In the lab, Aurora ran kubectl logs --previous 15 times across the 18 crash-loop investigations and got unable to retrieve container logs 4 times (01 r1, 02 r2, 09 r2, 12 r1, counted from each run's steps-full.json). The state we captured before the graded investigations shows the pattern: of 22 --previous captures for looping containers, the 9 that failed were exactly the 9 whose current state was Terminated (the 8 containers OOM-killed within a second of starting in 01, 02 and 12, and the nginx container in 05 r2), and all 13 taken while the container was Waiting or running again returned the crash log. In 12 r1 Aurora did what we recommend: after --previous failed it ran plain kubectl logs on the same pod and got worker started, batch size 2500. If you get that message, drop --previous. Pass -c <container> in multi-container pods.

3. Read the events. kubectl describe pod <pod> ends with them. A Killing event means the kubelet stopped the container on purpose; no Killing event means the process exited or the kernel killed it.

Now match the output to one of the cases below.

OOMKilled, exit 137: killed at the memory limit (scenarios 01, 02, 12, 17)

Scenario 01 raised a worker's BATCH_SIZE from 500 to 2500 against a 192Mi limit. kubectl -n shop describe pod, captured before the investigation (v0-run2, r1, trimmed to the container):

    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 02 Oct 2026 12:25:50 -0400
      Finished:     Fri, 02 Oct 2026 12:25:50 -0400
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 02 Oct 2026 12:25:27 -0400
      Finished:     Fri, 02 Oct 2026 12:25:27 -0400
    Ready:          False
    Restart Count:  3
    Limits:
      memory:  192Mi
    Requests:
      cpu:     50m
      memory:  64Mi
    Environment:
      BATCH_SIZE:  2500

Started and Finished in the same second means the limit is too small for what the process does at startup, so look at what changed in the spec (here the env var). In scenario 17, a slow leak, the same reason came after the container had run from 16:33:11 to 16:35:31, and --previous showed 28 normal processed batch of 500 items lines before it died: same reason, different fix. Both cases are in OOMKilled and memory leak OOMKilled.

Error, exit 137, and a Killing event: the liveness probe (scenario 06)

The api's liveness probe was moved from port 8080 to 8081, where nothing listens (v0-run2, r1, pod api-c7b987b4d-9knph; trimmed: the Ready, Limits and Requests lines, the events header and the first five events are cut):

    State:          Running
      Started:      Fri, 02 Oct 2026 12:31:50 -0400
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Fri, 02 Oct 2026 12:30:40 -0400
      Finished:     Fri, 02 Oct 2026 12:31:50 -0400
    Restart Count:  1
    Liveness:   http-get http://:8081/healthz delay=15s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8080/healthz delay=0s timeout=1s period=5s #success=1 #failure=3
Events:
  Warning  Unhealthy  9s (x6 over 99s)    kubelet            spec.containers{api}: Liveness probe failed: Get "http://10.244.2.14:8081/healthz": dial tcp 10.244.2.14:8081: connect: connection refused
  Normal   Killing    9s (x2 over 79s)    kubelet            spec.containers{api}: Container api failed liveness probe, will be restarted

137 with Reason: Error, not OOMKilled: the kubelet sent the kill, not the kernel's memory killer. The previous log is a healthy app serving 200 on /healthz and /items until it was killed. The fix is the probe port, never removing the probe: liveness probe failed.

Error, exit 1, and a config error: the app refused its config (scenario 05)

A ConfigMap edit turned proxy_connect_timeout into proxy_conect_timeout. Last State: Terminated, Reason: Error, Exit Code: 1, and the end of the crashed nginx's log (v0-run2, r1):

/docker-entrypoint.sh: Configuration complete; ready for start up
2026/10/02 16:30:48 [emerg] 1#1: unknown directive "proxy_conect_timeout" in /etc/nginx/conf.d/default.conf:9
nginx: [emerg] unknown directive "proxy_conect_timeout" in /etc/nginx/conf.d/default.conf:9

The log names the file and line. nginx unknown directive has the fix and how to test a config before it ships.

Error, exit 1, and a dependency: three causes, one exit code (scenarios 09, 10, 16)

Three different faults produced the same Reason: Error, Exit Code: 1 and the same last line, because the api gives up after five tries to reach redis. Only the error on the lines above it differs.

Scenario 10, REDIS_HOST set to redis-master, a Service that does not exist (v0-run2, r1, --previous):

redis connection failed (attempt 1/5): [Errno -2] Name does not resolve
redis connection failed (attempt 2/5): [Errno -2] Name does not resolve
redis connection failed (attempt 3/5): [Errno -2] Name does not resolve
redis connection failed (attempt 4/5): [Errno -2] Name does not resolve
redis connection failed (attempt 5/5): [Errno -2] Name does not resolve
giving up: redis unavailable

Scenario 16, the right host name but a pod dnsConfig with the wrong search domain: the identical six lines (v1-run1, r1). Scenario 09, redis stuck Pending on a PVC whose StorageClass does not exist (v0-run2, r1):

redis connection failed (attempt 1/5): [Errno 111] Connection refused
redis connection failed (attempt 2/5): [Errno 111] Connection refused
redis connection failed (attempt 3/5): [Errno 111] Connection refused
redis connection failed (attempt 4/5): [Errno 111] Connection refused
redis connection failed (attempt 5/5): [Errno 111] Connection refused
giving up: redis unavailable

Name does not resolve means the name is wrong or the pod's DNS settings are; compare the env var with kubectl get svc, then the pod's dnsPolicy and dnsConfig. Connection refused means the name resolved to a Service with nothing behind it; go to the dependency's own pods (here redis-5fff65475c-t2qf5 0/1 Pending). In all three, redis or CoreDNS restarts would change nothing, and the crashing pod is the messenger. Runbooks: Name does not resolve, dnsPolicy None, PVC Pending.

Error, exit 127: command not found (reproduction)

Not in the lab scenarios, so we reproduced it: command: ["sh", "-c", "exec start-server --port 8080"] in a busybox image that has no start-server. Last State: Terminated, Reason: Error, Exit Code: 127, and the log (logs-1.txt):

sh: exec: line 0: start-server: not found

127 is the shell's code for a command it could not find. Usual causes: a typo, a binary not on PATH in a new base image, a script that needs a shell the image lacks. Fix the command or the image.

StartError, exit 128: the runtime could not start the process (reproduction)

The same mistake without a shell, command: ["/app/server"], fails earlier. There is no log, because no process ever ran; the reason is in the termination message (describe-repro.txt, trimmed):

    State:          Waiting
      Reason:       RunContainerError
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "/app/server": stat /app/server: no such file or directory
      Exit Code:    128
      Started:      Wed, 31 Dec 1969 19:00:00 -0500
      Finished:     Thu, 29 Oct 2026 12:14:07 -0400
    Ready:          False
    Restart Count:  9

A Started time in 1969 is Unix time zero shown in local time: the process never started. kubectl logs returns nothing. Check the path in the image, the image architecture, and the file's execute bit.

Completed, exit 0: a one-shot task under a Deployment (reproduction)

sh -c "echo migration done; exit 0" in a Deployment. It succeeds, and loops anyway (describe-repro.txt, trimmed):

    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 29 Oct 2026 12:09:03 -0400
      Finished:     Thu, 29 Oct 2026 12:09:03 -0400
    Ready:          False
    Restart Count:  9

The Deployment documentation says "Only a .spec.template.spec.restartPolicy equal to Always is allowed", so a process that finishes is restarted and backed off like one that failed. Run one-shot work as a Job (or an init container), or make the process stay in the foreground.

What is the fix?

Fix the cause the steps above found, in the runbook linked for it. A few rules hold for all of them.

After the fix, a pod deep in the backoff waits up to 5 minutes before its next try. Deleting that pod (kubectl delete pod <pod>, the ReplicaSet creates a new one) starts a new container at once with a fresh backoff: our exit 1 pod was at back-off 5m0s, and its replacement started within a second of the delete and was retried 15 seconds later (delete-pod.txt). Only do that after the cause is fixed, or the new pod loops too. Risk: the pod is unavailable until the new one is Ready, which for a pod that is already crash looping changes nothing.

How do you verify it worked?

Pass condition: RESTARTS stops climbing, the container is Running and Ready, and it stays that way for more than 10 minutes, the point at which the kubelet resets the backoff.

kubectl -n <namespace> get pods -l app=<app> -w
kubectl -n <namespace> get pod <pod> -o jsonpath='{.status.containerStatuses[0].restartCount} {.status.containerStatuses[0].state}'

For a rollout, also kubectl rollout status deploy/<name> should report the rollout finished, and the old ReplicaSet should scale to 0. For a liveness case watch a little longer: our 06 pods ran 70 seconds between kills, so "it is Running" is true for most of every cycle.

How do you prevent it?

Alert on restarts as well as the waiting reason. The KubePodCrashLooping rule in kubernetes-mixin, as of today on its master branch, is the following, shown with the default kube-state-metrics selector filled in and the optional namespace selector empty (the source has template placeholders for both):

max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff", job="kube-state-metrics"}[5m]) >= 1

with for: 15m. It only sees the time a container spends waiting with that reason, which after each crash comes only once the container has sat terminated for a while, and none while a liveness-killed container is running again. Once the delay reaches 5 minutes each cycle has a waiting stretch, so the 5 minute max_over_time window should bridge the gaps and the rule should fire on a sustained loop; we did not run Prometheus against the reproductions, so that part is inference. Either way it fires late. A second rule on the restart counter covers the early minutes and the liveness pattern:

increase(kube_pod_container_status_restarts_total{job="kube-state-metrics"}[15m]) > 3

Tune the window and threshold to your workloads; a Job that retries by design will trip it.

Make the reason survive the log. Set terminationMessagePolicy: FallbackToLogsOnError on containers. Per the termination message docs, Kubernetes then uses "the last chunk of container log output if the termination message file is empty and the container exited with an error", up to 2048 bytes or 80 lines. With it set, the reproduction's crash reason is in the pod status, where describe and anything else that reads the pod can see it without the log (fallback.txt):

{"containerID":"containerd://9173ef60643217c709e3fe67658a32ba5f9940095db3c5261957321b13aedf3e","exitCode":1,"finishedAt":"2026-10-29T15:54:30Z","message":"starting\nfatal: config file /etc/app/app.yaml not found\n","reason":"Error","startedAt":"2026-10-29T15:54:30Z"}

It worked for the OOMKilled container too: "message":"loading cache\n","reason":"OOMKilled" (triage-columns.txt). That is the fix for the unable to retrieve container logs problem above.

Test. Start the image in CI with the production config and env (a kubectl run or a plain docker run with the same command) and fail the build if it exits within 30 seconds. The startup OOM (with the same memory limit), the nginx config error and the three reproduced command mistakes would all have failed that check; the probe, the dependency faults and the slow leak would not.

Common mistakes

• Searching for the word. kubectl get pods | grep CrashLoopBackOff would have missed six of our seven looping reproduction pods at the moment we listed them. Filter on restarts.
• Treating 137 as OOM. Scenario 06 exits 137 with Reason: Error; it is a liveness kill and more memory changes nothing.
• Trusting --previous when it says nothing. unable to retrieve container logs is not "there were no logs". Run kubectl logs without it, or read the termination message.
• Fixing the messenger. In scenarios 09, 10 and 16 the api looped, and the faults were a PVC, an env var and a pod DNS setting. Restarting or scaling the api did nothing in any of them.
• Removing the liveness probe to stop the restarts. It stops these restarts and the next deadlock's.
• Deleting the old ReplicaSet's pods during a stuck rollout. They are the only ones serving.
• Waiting after the fix. A pod at back-off 5m0s sits idle for up to five minutes with a correct config. Delete it once, after the fix.

What we saw in the lab

The Aurora lab is a five-component shop app on a three-node kind cluster, broken one way per scenario. Nine of the 19 scenarios in the benchmark produce a restarting container. For each of them the lab runner posted a fixed KubePodCrashLooping alert text from the scenario file to Aurora; no Prometheus rule evaluated anything, so these runs say nothing about alert timing. Aurora ran kubectl under its default observability_only command policy, which blocks mutating commands, two runs per scenario, each in a fresh organization. Verdicts are from a judge model (us.anthropic.claude-opus-5-5), and every one below was read by a person at Arvo (from benchmark/results.json).

16 of 18 correct, 2 partial, none wrong. Both partials are scenario 12, and neither missed the crash loop: Aurora found the OOM and the BATCH_SIZE change in both runs, then judged the rest of the namespace from pod status alone: r1 wrote that the api and frontend "remained healthy and unaffected", r2 that they "were running normally and unaffected". Neither run looked at a Service, and the second fault was the frontend Service selecting no pods. A crash loop is loud enough to end an investigation early. That is its own runbook: a second fault the alert did not cover.

What surprised us:

• The exit code was the same for unrelated causes. Four scenarios exited 1 and three of them printed the same last line. The errno one line up (Name does not resolve or Connection refused) split them. A table that maps "exit code 1" to one cause is wrong.
• --previous failed often. Four of Aurora's 15 --previous calls returned unable to retrieve container logs, and so did all 8 of our own captures for the fast OOM kills (9 of 22 captures in all). Aurora's 01 r1 report listed the previous logs under "Not Checked" and diagnosed from describe instead, which was right; in 01 r2 the same call succeeded and returned the one line that matters, worker started, batch size 2500; in 12 r1 Aurora got that line from plain kubectl logs after --previous failed. Our reproductions today showed the same thing for exit 1, 127, 0 and OOM containers.
• 137 did not fool it. In both 06 runs Aurora read Reason: Error, Exit Code: 137 as a kubelet kill and ruled out OOM from measured memory (11 to 14Mi in r1, 12 to 18Mi in r2, against a 128Mi limit).
• The slowest run was not about the crash. 09 r1 took 406 seconds, more than any other single-fault run here, and still reached the right cause.

The extra cases on this page (exit 127, 128, 0, the backoff timing and the STATUS sampling) are reproductions on our own cluster, rcrash, on 2026-10-29. Aurora has not been run on them.

How Aurora handles this

Aurora receives the alert, connects with the kubeconfig you give it, and runs the same reads shown here: pods, describe, logs of the crashed instance, events, the ReplicaSet diff. It cites every command output in its report and lists what it could not check, such as logs that were unavailable. It does not restart or patch anything; mutating commands are blocked by the default command policy, and it suggests the fix for a person to apply. On the nine crash-loop scenarios above it named the crash cause in all 18 runs and missed a second, unrelated fault in two. The traces for every run are in the lab repository.

• Start free: aurora-ai.net (hosted, no infrastructure to run)
• GitHub: github.com/Arvo-AI/aurora
• Book a demo: cal.com/arvo-ai/demo

Related: OOMKilled, liveness probe failed, nginx unknown directive, Name does not resolve, ImagePullBackOff, CreateContainerConfigError, the lab benchmark, paste an error into the diagnose tool, and root cause analysis for SREs.

Sources and verification

• Lab outputs: evidence-before/ (cluster state captured before each investigation) and steps-full.json (Aurora's tool calls) in runs/v0-run2/{01,02,05,06,09,10} and runs/v1-run1/{12,16,17}, 2026-10-02; look-alikes from runs/v0-run2/03-bad-image-tag, 04-missing-secret and runs/v1-run1/18-init-waits-missing-service. Faults injected with each scenario's break.sh.
• Reproductions: kind cluster rcrash from the lab's cluster/kind.yaml, 2026-10-29. Kubernetes v1.37.0, kind v0.33.0, containerd 2.3.4, kubectl v1.35.3, busybox:1.37, python:3.12-alpine. Files in runs/search-2026-10-29/crashloopbackoff-repro/: backoff-poll.tsv, state-samples.tsv, 06-liveness-poll.txt, logs-1.txt, triage-columns.txt, fallback.txt, versions.txt, and the manifests. The backoff and STATUS timings come from one kind cluster on a laptop Docker VM that was shared with several other kind clusters while the disk was nearly full; the 65 to 86 second lag before the state turned waiting did not show on the lab cluster on 2026-10-02 and is probably that environment.
• Aurora commit 87d22baf, single agent, RCA model bedrock/us.anthropic.claude-sonnet-5; verdicts from benchmark/results.json.
• Kubernetes documentation, read 2026-10-29: Pod lifecycle (backoff, reset, CrashLoopBackOff state, maxContainerRestartPeriod, ReduceDefaultCrashLoopBackOffDecay), Images (ImagePullBackOff), Determine the reason for pod failure (FallbackToLogsOnError), kubectl logs (--previous).
• Alert rule: kubernetes-mixin apps_alerts.libsonnet, read 2026-10-29.