19 Kubernetes faults. 3 agents. Every trace public.

How a run works

1. kind v0.33.0, Kubernetes v1.37.0, 3 nodes. The shop app: nginx, a Python API, a worker, redis, a load generator. Synthetic, no customer code.
2. One fault is injected with the scenario's break.sh; check.sh proves it is visible.
3. Each agent gets the alert text. Aurora's prompt adds "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause."; HolmesGPT's adds "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." K8sGPT has no notion of an alert and scans the namespace. All three got the same kind admin kubeconfig, each behind its own guard: Aurora's default read-only observability_only command policy, HolmesGPT's command allow-list, and K8sGPT's read-only analyze. No transcript shows a mutating command running.
4. HolmesGPT ran through holmes ask with default toolsets and a 40-step cap it never reached; the lab has no Alertmanager, so holmes investigate was not used. Its own command allow-list blocked kubectl rollout history in 12 runs, 2 of its 7 non-correct runs among them; we did not rerun those with the command allowed. Aurora's policy had the same block in the first lab report.
5. Correct means every must_identify item in the scenario is stated and no listed wrong answer is asserted; partial means some items. Judge: us.anthropic.claude-opus-5-5, the same vendor as every model under test. The prompt does not name the agent, though the tool names in each command list give it away.

Every tool ran twice per scenario. Aurora's 38 verdicts were checked by a person, who changed 0 of them. HolmesGPT’s and K8sGPT’s were graded by the judge model, read over by a second model (which agreed on all 76), then re-graded from scratch by a third. The third agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels, where K8sGPT only restated the symptom: the judge said partial or wrong, the reviewer said no answer. We kept the judge’s labels, and every grade file carries both opinions. No person has checked these 76 yet. As a bias check, every answer was graded again with the agent’s name, its tool names and the cluster name removed from the judge’s prompt (runner/grade_blind.py). The blind verdict matched the published one for 38 of 38 Aurora, 37 of 38 HolmesGPT, 36 of 38 K8sGPT runs; blind correct counts are Aurora 34, HolmesGPT 30, K8sGPT 0. Every disagreement is listed in runs/blind-regrade.md in the repo. Aurora's runs are from commit 87d22baf, the day before the scenarios were written, so it was not tuned on them. The fixes the first lab report prompted are open pull requests on the Aurora repo; the six reruns on the fixed build are in the lab repo and not in this table.

What is not comparable: K8sGPT's row. Its analyzers flag objects and the model explains the flag, so it is scored on a criterion built for alert-driven investigators. It ran on Sonnet 4.5 because it sends a temperature parameter that Bedrock rejects for Sonnet 5; we did not try a proxy that strips it. Times: Aurora's include contention from its own background jobs in the lab stack (REPORT.md in the repo); K8sGPT's cover two analyze passes, JSON then text. Costs are recorded for Aurora only; HolmesGPT's token counts were not captured.

Rerun it, or add your agent

Everything is in Arvo-AI/aurora-lab: scenarios, the runner, every transcript and grade, and this table as results.json.

git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab
./lab.sh up && ./lab.sh break 03-bad-image-tag
./runner/h2h.sh 03-bad-image-tag runs/mine

To put another agent on the board, open an issue titled "Benchmark: <agent>" with a script that takes a kubeconfig and the alert text and prints the root cause. We run it on all 19 faults and publish the result, whatever it is. Submit an agent.

Read the full write-up: lab report v1. Fix the faults yourself: runbooks, each with Aurora's real trace.