An LLM diagnostic agent you could actually run at a bank

The 3 a.m. problem

Root-cause analysis on Kubernetes is a ritual every on-call engineer knows: a pod is crash-looping, so you describe it, read the events, pull the logs, check the node, check recent deployments, check resource limits, and somewhere between the fourth and the ninth kubectl command, the picture forms. It's not hard. It's just slow, repetitive, and it happens at exactly the hours when humans are worst at it.

This is, on paper, a perfect job for an LLM agent: the investigation is a loop of "gather evidence, form hypothesis, gather more evidence," and the evidence is all text. So I built one — eks-diagnose-agent, a Go binary that connects to a cluster, investigates a failing workload the way an engineer would, and writes up a root-cause report.

Building the happy path took a fraction of the total time. The rest went into a question that the demos you see on social media never touch: what would it take for a bank's security review to let this thing anywhere near a production cluster?

What the agent does

The core is an agentic tool-use loop. The model doesn't get shell access; it gets a small set of typed, purpose-built tools — list pods, describe a resource, fetch recent logs, read events, summarize node conditions. Each turn, it picks a tool, receives structured output, and updates its working hypothesis until it's confident enough to write the report.

Go was a deliberate choice: a single static binary, no runtime dependency tree to explain to a security reviewer, and the Kubernetes client libraries are first-class. The output is a report for a human — the agent diagnoses; it does not remediate. That line is load-bearing, and we'll come back to it.

The uncomfortable part: your cluster talks back

Here is the threat model that most "AI ops" tooling quietly ignores. An LLM agent reading cluster state is consuming attacker-influenceable input. Pod logs are written by workloads — including compromised ones. Annotations, labels, container args, event messages: all of it is text that somebody other than you may control, and all of it flows into the model's context.

Which means a compromised workload doesn't need to attack your agent's code. It can attack its reasoning:

If your agent has write access and treats model output as commands, that log line just gave an attacker a lever on your infrastructure — laundered through an AI that your own team deployed. This is prompt injection, and in a cluster context the injection surface is everywhere.

The defenses

The design answers that threat model in layers, and none of the layers is "we prompt it really carefully." Prompts are guidance; boundaries have to live outside the model.

• Read-only by construction. The agent's ServiceAccount has a purpose-built ClusterRole with get/list/watch on the resources it needs and nothing else. No create, no patch, no delete, no exec, no port-forward. Even a fully hijacked reasoning loop has no verbs to act with.
• Allowlisted, typed tools. The model can't compose arbitrary API calls — it chooses from a fixed menu of tools with validated, typed parameters. There is no "run this kubectl string" tool, because that tool is a shell with extra steps.
• Untrusted data is fenced. Cluster-derived text (logs, events, annotations) is delimited and labeled as data in the context, and the system instructions explicitly refuse instructions that arrive inside it. This is a mitigation, not a guarantee — which is why it's the third layer, not the first.
• Output is a report, not an action. The agent's only side effect is a written diagnosis. Remediation stays with a human who can smell a suspicious recommendation — the same separation of duties a bank already applies to change management.
• Injection is tested, not assumed away. The repo ships with a deliberately hostile test pod that writes injection payloads into its own logs, labels, and events. The test suite asserts the agent's report stays on-topic and its tool sequence stays within policy while investigating it. If you deploy an agent without an adversarial workload in your test matrix, you don't know what you've deployed.

Supply chain, because auditors will ask

A security review of an AI tool doesn't stop at the AI. The repository carries a SECURITY.md that states the threat model in plain language and defines the scan surface: dependency scanning on the Go module graph, image scanning on the published container, and a pinned, minimal base image. A single static binary keeps the SBOM short — and a short SBOM is the most underrated security feature an internal tool can have.

None of this is exotic. That's the point: the controls that make an agent acceptable in a bank are the same controls the bank already applies to everything else — least privilege, separation of duties, adversarial testing, supply-chain hygiene. LLMs don't get an exemption, and they don't need special magic either.

The bank-grade checklist

If you're evaluating (or building) an LLM agent for a regulated cluster, this is the short version of what I'd hold it to:

• RBAC that makes the worst case boring — read-only, enumerated resources, no exec
• A closed tool set with typed parameters; no free-form command execution
• All cluster-derived text treated as untrusted; instructions inside it refused
• Human-in-the-loop for anything that changes state
• An adversarial (injection) workload in the test suite
• SECURITY.md, dependency & image scanning, pinned minimal base image
• Works where banks work: private networking, and a story for air-gapped or proxy-only environments

The interesting frontier isn't whether LLMs can help operate Kubernetes — they clearly can. It's whether we build them with the same engineering discipline we demand from everything else in the cluster. In a bank, that's not a nice-to-have. It's the entry ticket.