In partnership with
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | July 24, 2026
The Promise: By the end of this issue you will understand how to build an AI troubleshooting agent where deterministic code collects evidence, the LLM only reasons, and rule engines override the LLM's own confidence claims.
The Fence: This issue covers the architecture, the confidence validation pipeline, and the LLM-as-judge layer of DevOps Open Agent. It intentionally excludes the frontend implementation, deployment hardening, and any comparison of underlying LLM quality across providers.
What It Actually Does
Strip the marketing and DevOps Open Agent is a self-hosted Docker Compose stack (FastAPI backend, Next.js 15 frontend) that runs read-only investigations against your infrastructure, then asks an LLM to explain what it found.
The concrete facts:
Four core agent modules: Kubernetes Debugging, AWS DevOps, Cloud Cost Detector, and PR Reviewer, with Performance Debugging and security scanning landing in v2
One shared LLM layer with five providers: OpenAI, Anthropic, Ollama, OpenRouter, and Gemini. Set
LLM_PROVIDER=ollamaand the entire platform runs air-gapped on local modelsRoughly 22,000 lines of Python backend and 14,000 lines of TypeScript frontend, Apache 2.0 licensed
Storage is deliberately split: PostgreSQL for auth only, SQLite for every investigation record
Ships with 5 controlled Kubernetes failure scenarios (
tests/scenarios/) so you can validate the agent against known-bad clusters
It is a young project, low double-digit stars, 22 commits on main, built in public by Prashant Lakhera. What makes it worth dissecting is not scale. It is the discipline of where the AI is allowed to exist.
The Architecture, Unpacked
The design principle running through the entire codebase: evidence collection is deterministic code, reasoning is the LLM, and the LLM's output is then post-processed by rule engines that can override it.
Caption: Focus on the single choke point. Every agent module funnels through one shared AI pipeline, and the LLM's answer passes through a deterministic ConfidenceEngine before a human ever sees it.
Three decisions matter most, ranked:
Decision one: the LLM never touches infrastructure. Inspectors (pod_inspector, network_inspector, logs_collector, CloudTrail and CloudWatch collectors) gather evidence with plain Python and kubectl or boto3 calls. The LLM receives a structured JSON evidence bundle and returns a structured JSON diagnosis. The architecture doc states it plainly: never auto-execute remediation, human approval required.
Decision two: one LLM factory, five providers, zero provider lock-in. All modules call LLMProviderFactory.create(). Swapping Claude for a local Gemma model on Ollama is a one-line .env change plus a container restart. This is the entire vendor-neutrality story, implemented in 78 lines.
Decision three: post-hoc validation stacks. The raw diagnosis passes through a MultiPodEnricher, a ConfidenceEngine, a FixRecommendationEngine, and optionally an LLM-as-judge that scores the first LLM's work on five axes including command safety. The AI answer is treated as a draft, not a verdict.
The Code, Annotated
Snippet one: the ConfidenceEngine that overrules the model
This is the most interesting file in the repo. The LLM self-reports a confidence score. This engine audits that claim against the evidence the model actually cited.
# backend/app/ai/confidence_engine.py
class ConfidenceEngine:
"""Validate and adjust AI confidence scores based on evidence."""
VALID_SOURCES = {"logs", "events", "pods", "deployments",
"network", "topology", "observability"}
def apply(self, diagnosis: DiagnosisResult, evidence_context: dict) -> DiagnosisResult:
score = max(0, min(100, diagnosis.confidence_score))
# Count only evidence sources the model actually cited AND that exist
evidence_sources = {
item.source for item in diagnosis.evidence
if item.source in self.VALID_SOURCES
}
available_signals = self._count_available_signals(
evidence_context.get("investigation", {})
)
# ← THIS is the trick: a 95% confident answer with thin citations
# gets forcibly demoted. The LLM's self-belief is untrusted input.
if score >= 90 and len(evidence_sources) < 2 and available_signals >= 2:
score = 79 # confident but under-cited → knocked below the 80 bar
elif score >= 70 and len(evidence_sources) == 0:
score = 35 # confident with ZERO citations → flagged as unreliable
diagnosis.needs_more_data = True
elif score <= 39 and available_signals >= 2 and len(evidence_sources) >= 2:
score = 55 # timid but well-evidenced → nudged up
diagnosis.confidence_score = score
return diagnosis
Caption: Deterministic rules clamp the LLM's self-reported confidence. A 90+ score with fewer than two cited evidence sources becomes 79. A 70+ score with zero citations collapses to 35 and triggers a needs-more-data flag.
Snippet two: LLM-as-judge with a command safety axis
A second, independent LLM call reviews the first diagnosis. Note axis C: the judge explicitly audits whether the recommended kubectl commands are safe to hand a human.
# backend/app/ai/judge.py (system prompt, abbreviated)
_SYSTEM_PROMPT = """You are an expert SRE peer-reviewer acting as a judge
for an AI-generated Kubernetes diagnosis.
Evaluate the diagnosis on five axes:
A. Factual consistency - does every claim match the evidence?
B. Evidence grounding - is the root cause supported by cited sources?
C. Command safety - are kubectl commands safe (no destructive # ← THIS axis
deletes, no broad-scope ops without # is the part
namespace scoping)? # most agent
D. Completeness - did the diagnosis miss signals? # frameworks
E. Actionability - is the fix specific enough to act on? # skip entirely
Respond with ONLY JSON:
{ "verdict": "agree | partially_agree | disagree",
"confidence_score": <0-100>, ... }
Do NOT invent evidence that is not present in the investigation data."""
Caption: The judge is not grading writing quality. It is auditing factual grounding and whether the suggested commands could hurt a production cluster.
The primary diagnosis call itself runs at temperature=0.1 in root_cause_analyzer.py, and if the provider errors out, a fallback diagnosis with confidence_score=0 is returned rather than a retry loop. Fail visibly, not silently.
It In Action
The repo ships reproducible failure scenarios. Here is scenario 01 end to end.
Step one, the exact input. Apply the bundled manifest, which deploys a payment-api pod designed to crash because DATABASE_URL is missing:
# tests/scenarios/01-crashloop-missing-env.yaml (excerpt)
containers:
- name: payment-api
image: busybox:1.36
command: ["/bin/sh", "-c"]
args:
- |
if [ -z "$DATABASE_URL" ]; then
echo "Error: DATABASE_URL environment variable is missing"
exit 1 # ← guaranteed CrashLoopBackOff, on purpose
fi
sleep 3600
Caption: A controlled CrashLoopBackOff. The pod exits with code 1 every restart, generating logs, events, and pod-status evidence simultaneously.
Step two, trigger the investigation:
kubectl apply -f tests/scenarios/01-crashloop-missing-env.yaml
curl -X POST http://localhost:8000/api/v1/investigate \
-H "Content-Type: application/json" \
-d '{"cluster_id":"kind-devops-agent","include_ai":true}'
Step three, the pipeline runs. Discovery finds the kda-test-crashloop namespace. The pod inspector flags payment-api with restart count climbing and state CrashLoopBackOff. The logs collector captures the literal error line. The events analyzer records BackOff events. Three independent evidence signals now exist: pods, logs, events.
Step four, the exact output shape. The LLM must return this schema (from prompt_builder.py), and for this scenario the diagnosis lands in the top confidence band because the system prompt defines 90 to 100 as "multiple evidence sources confirm the same cause":
{
"root_cause": "Container payment-api exits with code 1 because the DATABASE_URL environment variable is not set",
"evidence": [
{"source": "logs", "detail": "Error: DATABASE_URL environment variable is missing"},
{"source": "pods", "detail": "payment-api in CrashLoopBackOff, restartCount rising"},
{"source": "events", "detail": "BackOff restarting failed container"}
],
"suggested_fix": "Add DATABASE_URL to the deployment env spec",
"kubectl_commands": ["kubectl -n kda-test-crashloop set env deployment/payment-api DATABASE_URL=<value>"],
"confidence_score": 92,
"needs_more_data": false
}
Step five, the audit. ConfidenceEngine checks: score 92, three cited sources, three available signals. The 92 survives because citations are dense. Had the model cited only one source, the exact same answer would have been published at 79. The judge then issues a verdict of agree because the kubectl command is namespace-scoped and non-destructive. The result persists to SQLite with agent_type=kubernetes and shows in the history UI with root cause, confidence, and timestamp.
Same crash, same model, different citation density: different published confidence. That is the whole design in one number.
Why This Design Works, and What It Trades Away
Why it works. The failure mode of AI troubleshooting tools is not wrong answers, it is wrong answers delivered confidently. By separating collection (deterministic), reasoning (LLM), and validation (deterministic plus a second LLM), each layer can fail independently and visibly. The provider factory means the reasoning layer is a commodity: teams with compliance constraints run Ollama locally and lose zero platform features. The SQLite-for-investigations choice keeps the entire history portable in a single file, ideal for a self-hosted tool where the operator owns the data.
What it trades away. Plenty. Rule-based confidence clamping is crude: a genuinely correct single-source diagnosis gets punished to 79 just for citing one source. The judge doubles LLM cost and latency per investigation. SQLite caps concurrent write throughput, fine for a team, wrong for a fleet. The frontend bakes NEXT_PUBLIC_API_BASE_URL at build time, so changing the public URL means rebuilding a container, a classic self-hosting paper cut. And the whole platform is read-only by design: it will tell you the fix but never apply it, which is a feature for trust and a limitation for automation-hungry teams.
Technical Moats
Honest assessment: the code is not the moat. Any senior engineer could rebuild the pipeline in weeks. The structural advantages are elsewhere:
The failure scenario corpus. Shipping controlled, reproducible broken-cluster manifests (CrashLoopBackOff, ImagePullBackOff, OOMKilled, FailedScheduling, selector mismatch) turns "does the AI work" from vibes into a regression test. Growing that corpus into hundreds of scenarios is accumulated discipline competitors would have to grind out.
Distribution flywheel. The maintainer runs a 100 Days of GenAI for DevOps course with 6.5K Medium followers and ships features against community feedback weekly, five major features in seven days in one stretch. The product is the course material and the course is the funnel.
The self-hosted plus local-LLM position. Datadog, PagerDuty AIOps, and most AI SRE startups cannot offer "your infra, your LLM, your data never leaves." An Apache 2.0 stack that runs entirely on Ollama occupies ground the incumbents structurally cannot take.
Insights
Insight One: the AI is the least differentiated part of an AI DevOps agent. The community obsesses over which model diagnoses clusters best. This codebase suggests the opposite: the model is a swappable 78-line factory, while roughly 22,000 lines go to evidence collectors, schema contracts, validation engines, and storage. The moat in agentic infrastructure tooling is the deterministic scaffolding around the model, not the model. If your "AI agent" architecture collapses when you swap GPT for a local Gemma, you built a prompt, not a platform.
Insight Two: LLM self-reported confidence should be treated as adversarial input, and almost nobody does. LangChain-style agent stacks pass model confidence straight to the user. DevOps Open Agent clamps it with hard rules and then cross-examines it with a second model whose rubric includes command safety. This inverts the industry default, where guardrails filter inputs but outputs ship raw. For any agent that recommends actions against production systems, output validation is the guardrail that matters.
Takeaway
A diagnosis the model rates at 95% confidence gets published at 35 if it cited zero evidence sources. The exact same root cause text, demoted by 60 points, purely because the citations were missing. Confidence in this system is not a measure of the model's belief, it is a computed function of citation density.
TL;DR For Engineers
Four agent modules (K8s, AWS, cloud cost, PR review) share one 78-line LLM factory covering OpenAI, Anthropic, Ollama, OpenRouter, and Gemini
Evidence collection is 100% deterministic Python; the LLM only reasons over a structured JSON bundle at temperature 0.1
A rule-based ConfidenceEngine overrides LLM confidence: 90+ with under two citations becomes 79, 70+ with zero citations becomes 35
An optional LLM-as-judge re-audits every diagnosis on five axes, including whether kubectl commands are destructive
The platform never executes fixes. Read-only investigation, human-approved remediation, full history in a single SQLite file
Explain It Like I'm New
When something breaks in a company's cloud infrastructure, an engineer gets paged and starts detective work: reading logs, checking dashboards, running commands, and piecing together what went wrong. It is slow, stressful, and depends heavily on experience. The obvious idea is to let AI do the detective work. The problem is that AI models sometimes state wrong answers with total confidence, and a confidently wrong answer about production infrastructure is worse than no answer.
DevOps Open Agent handles this with a division of labor. Ordinary, predictable software gathers the facts: which services are failing, what the error logs say, what changed recently. Only then does the AI get involved, and its only job is to read those facts and explain the likely cause, like a consultant handed a complete case file rather than one allowed to wander the building.
Then comes the unusual part. The system does not take the AI's answer at face value. It checks whether the AI actually pointed to real evidence for its conclusion, and if it did not, the system automatically downgrades how trustworthy the answer is labeled. A second AI even reviews the first one's work. And the platform never fixes anything on its own; it only recommends, and a human decides.
This matters because it is a template for using AI in high-stakes operations: let the AI reason, but never let it be the last word.
See It In Action
DevOps Open Agent Demo, Prashant Lakhera (YouTube): youtube.com/watch?v=3VT8MeSLt5s. A full recorded walkthrough of the platform showing investigations, structured root cause output, evidence, suggested fixes, and confidence scores across providers.
DevOps Open Agent v2 walkthrough, Prashant Lakhera (YouTube): youtube.com/watch?v=9HjWLyoHdng. Covers the expanded platform: six agents, AI root cause analysis, and the Slack, Microsoft Teams, and PagerDuty integrations. Useful for seeing how investigations flow into on-call workflows.
Linux Performance Debugging announcement, Prashant Lakhera (Medium): devopslearning.medium.com. Shows the newest agent module in practice, host-level performance debugging over passwordless SSH, and how it fits alongside the K8s and AWS modules.
Community Conversation
Prashant Lakhera (maintainer), "We don't need 10 AI agents. We need one DevOps agent": the founding thesis, arguing that general-purpose agents fail at production infrastructure and that DevOps needs one purpose-built investigate → analyze → recommend → track loop.
Prashant Lakhera, "What a week for DevOps Open Agent": five major features shipped in seven days driven by community feedback, a window into how fast solo open-source AIOps tooling can iterate versus enterprise vendors.
Prashant Lakhera, PagerDuty integration announcement: the community pushed back that the tool lacked enterprise integrations; the response was Events API v2 incidents with severity mapping, dedup keys, and cooldown-based alert fatigue protection. A useful debate on what "enterprise-grade" actually requires from open source.
The Untrusted Model Is The Right Model
The strongest claim in this issue: DevOps Open Agent is a better reference architecture for production AI agents than most funded agent frameworks, precisely because it assumes the model is wrong until the evidence proves otherwise. Deterministic collection, schema-locked reasoning, rule-clamped confidence, judged output, human-gated execution. That stack is boring, and boring is exactly what you want standing between an LLM and your production cluster.
References
DevOps Open Agent repository (Apache 2.0)
Summary
DevOps Open Agent is a self-hosted, provider-agnostic AI troubleshooting platform for Kubernetes, AWS, cloud cost, and PR review, built on a strict split between deterministic evidence collection and LLM reasoning. Its defining architectural insight is that LLM output, including self-reported confidence, is untrusted input to be clamped by rule engines and audited by a second model. It matters as a working reference for how AI agents should be structured before they are allowed anywhere near production infrastructure.
🚀 Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value
Join 10,000+ engineers getting deep technical breakdowns at www.snackonai.com
AI help, without the trust tax.
Most AI tools ask you to trade your data for intelligence. Norton Neo doesn't. It's the first safe AI-native browser built by Norton, and it gives you powerful built-in AI without handing your privacy over to get it. Search, summarize, and write with AI built directly into your browser. Your data stays yours. Your context stays private.
Built-in VPN, anti-fingerprinting, and ad blocking come standard. No add-ons. No setup. No compromises.
Fast. Safe. Intelligent. That's Neo.