In partnership with
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 25, 2026
The promise: by the end of this issue you will know how OpenSRE turns an alert into an evidence linked diagnosis, which three engineering choices decide whether that diagnosis can be trusted, and what the latest research says those choices get wrong.
The fence: we cover the OpenSRE repository at its October 23, 2026 head (public alpha, v0.1), its README and deployment guide, and two root cause analysis (RCA) papers: why agents fail at cloud RCA and Microsoft's study of LLM agents for RCA. We did not run the agent against live infrastructure; it requires a signed in account and connected integrations. Our numbers come from the source code and a replay of its budgeting rules.
What It Actually Does
OpenSRE, from Tracer, is an Apache 2.0 Python framework for AI site reliability engineering agents. You ask a question ("why is checkout-api slow?") or an alert arrives, and an LLM agent queries your observability, infrastructure and deploy tools in a loop until it can answer with linked evidence.
Property | Value |
|---|---|
Status | Public alpha, README labeled v0.1; ~13 commits a day in the week we sampled |
Adoption | ~12,400 stars and ~1,800 forks (OSS Insight) |
Integrations | 60+ advertised; 124 entries under |
LLM providers | Anthropic, OpenAI, Codex, Ollama, Gemini, OpenRouter, TrustedRouter, NVIDIA NIM, Bedrock |
Entry points | Interactive REPL, headless |
Tool inventory (our count) | 238 side effect declarations: 186 read only, 42 mutating, 9 external, 1 none |
Deployment | OpenSRE Cloud managed gateway, EC2 AMI + systemd, or any ASGI host via the Dockerfile |
Stated goal | An open reinforcement learning environment for incident response, "the SWE-bench for AI SRE" |
The ambition matters. SWE-bench gave coding agents a scalable training signal; production incidents have no equivalent, because distributed failures are slow, noisy and hard to replay. OpenSRE wants to be both the agent and that environment.
The Architecture, Unpacked
OpenSRE is a strictly layered codebase, enforced in CI with import contracts, wrapped around one think, call tools, observe loop.
Caption: Notice the two approval paths at the top and the budget rules in the middle. Those two pieces decide what the agent may do and what it can still remember.
Four design decisions do the work.
One: tools carry their own safety metadata. Every tool declares a side_effect_level (none, read only, mutating, external), whether it may run concurrently, and an evidence mapper that turns raw output into citable evidence. Read only tools can run in parallel; anything that changes state runs alone.
Two: the LLM is the only router. An architecture decision record from June 2026 removed every regex and keyword intent rule. Tool choice comes only from the system prompt and each tool's description, use cases and anti examples. Fewer silent misroutes, but every routing fix becomes a prompt edit.
Three: context is managed like a cache. Each tool result is capped at about 10,000 tokens, keeping the beginning and end (parity with OpenAI Codex, per the repo's own research note). When the transcript nears the model's window, whole tool results are swapped for a stub that says "call the tool again if you still need it."
Four: identifiers are masked reversibly. Pod names, clusters and account IDs become stable placeholders such as <NS_0> before any external LLM call, and are restored in the final answer.
The Code, Annotated
First, a real tool, from integrations/grafana/tools/grafana_logs_tool/__init__.py. The decorator is the contract between the integration and the agent.
@tool(
name="query_grafana_logs",
display_name="Grafana Loki",
source="grafana",
side_effect_level=SideEffectLevel.READ_ONLY, # ← decides every approval gate downstream
concurrency_safe=True, # read only, so it may run in parallel
evidence_mapper=_map_grafana_logs, # raw logs → citable evidence objects
description=(
"Query Grafana Loki log streams for one service_name (required). "
"Optionally narrow with execution_run_id or pipeline_name and a lookback window."
), # ← THIS is the router: no regex, only prose
use_cases=["Retrieving application logs from Grafana Loki during an incident", ...],
anti_examples=list(_GRAFANA_LOGS_ANTI), # tells the LLM when NOT to pick it
requires=["service_name"],
injected_params=_GRAFANA_RUNTIME_PARAMS, # credentials injected, never chosen by the LLM
is_available=_query_grafana_logs_available, # hidden unless Grafana is configured
)
def query_grafana_logs(service_name: str, execution_run_id: str | None = None, ...):
...
Caption: Routing quality equals description quality. With 124 integration folders, the tool catalog is effectively a prompt the whole community edits.
Second, the two rules that decide safety and memory, verbatim from surfaces/cli/ask/approval.py and core/context_budget.py.
_GATED_SIDE_EFFECTS = frozenset({SideEffectLevel.MUTATING, SideEffectLevel.EXTERNAL})
def approval_required(tool: RuntimeTool) -> bool:
if bool(getattr(tool, "requires_approval", False)):
return True
level = getattr(tool, "side_effect_level", None)
if level is None:
return True # ← fail closed: undeclared tools need approval
return level in _GATED_SIDE_EFFECTS
def _eviction_priority(exchange: _ToolExchange) -> tuple[int, int, int]:
"""Lower priority tuple is evicted first."""
duplicate_rank = 0 if exchange.duplicate_only else 1
return (duplicate_rank, -exchange.token_estimate, exchange.start)
# ↑ THIS is the trick: after duplicates, the LARGEST result goes first
Caption: The CLI fails closed on side effects. The eviction rule frees the most space per stub, which also means the biggest log dump is the first evidence to disappear.
Third, our replay of the budget rules. We copied the repo's constants (40,000 byte provider cap, 48,000 byte history cap, 0.5 estimated tokens per character, 25,000 token output reserve, largest first eviction) into a script and ran an 11 step investigation, the average run length in the OpenRCA study.
HIST_CAP = int(10_000 * 4 * 6 / 5) # 48,000 bytes per recorded tool result
TOK_PER_CHAR = 0.5 # OpenSRE's conservative estimator
OVERHEAD = 20_000 # our assumption: system prompt + tool schemas
# history cap: 48,000 bytes = 24,000 estimated tokens (~12,000 real at 4 bytes/token)
# bytes dropped by the per-result cap: 440,000 of 756,000 (58%)
# unknown model (128K): 3.5 full-size results fit; first eviction at step 7
# step 7: evicted 'search_logs_checkout' (from step 4)
# step 8: evicted 'get_traces_slow_spans' (from step 5)
# Claude (200K): 6.5 full-size results fit; step 11 evicts 'search_logs_checkout'
# Gemini (1M): no evictions
Caption: On a default 128K model the agent has stubbed its first log search by step 7, while the investigation is still running.
It In Action
Input: opensre ask "why is checkout-api slow?" with Grafana, Kubernetes and GitHub connected. The tool outputs below are our synthetic sizes, chosen to resemble a real incident; the rules applied to them are OpenSRE's.
Step one, gather. The agent reads the alert (3 KB) and loads the matching runbook (9 KB) by exact alert name, as the runbook skill requires.
Step two, fan out. It queries p99 latency metrics (14 KB), then Loki logs for checkout (220 KB raw). The log result is cut to its first and last ~20 KB before the model sees it; the middle 172 KB is reported as omitted.
Step three, correlate. Slow spans from traces (60 KB raw, capped), recent deploys (8 KB), Kubernetes events (70 KB raw, capped), payments logs (180 KB raw, capped). On a 128K window the budget overflows at step 7, and the checkout log search from step 4 is replaced by a stub.
Step four, conclude. After a config diff, a database connection query and a second checkout log search, the agent writes an answer linking each claim to evidence, masks restored, and posts it to Slack.
Output: an evidence linked diagnosis built from 756 KB of raw tool output, of which 58% never reached the model and four of eleven results had been stubbed by the end.
Model window | Ceiling (estimated tokens) | Full-size results that fit | Evictions in 11 steps |
|---|---|---|---|
Unknown model (128K default) | 103,000 | 3.5 | 4 |
Claude family (200K) | 175,000 | 6.5 | 1 |
Gemini family (1M) | 975,000 | 39.8 | 0 |
Caption: Model choice changes what the agent remembers. On small windows, the evidence it saw early may be gone when it writes the root cause.
Why This Design Works, and What It Trades Away
Why it works. The layered architecture keeps 124 vendor integrations from tangling into the agent core; CI rejects forbidden imports. Side effect metadata lets one registry serve an interactive shell, a CI job and a scheduled job with different safety rules. Head and tail truncation preserves error summaries and exit codes, which tend to appear at the ends of output. Reversible masking makes it possible to use hosted LLMs on production identifiers.
What it trades away.
Evidence in the middle. Head and tail truncation drops the middle of long logs, often where an intermittent error sits.
Evidence over time. Largest first eviction removes the biggest observation first, typically a log dump.
One safety model per surface. The CLI gates all mutating and external tools; scheduled jobs allow only read only tools; the chat gateway's approval buttons key on an explicit
requires_approvalflag, which only two tools intools/andintegrations/set (shell and Python execution). Teams running the gateway should confirm how their mutating tools are gated.Open source with a sign in. The interactive shell only opens for an active account and first launch activates a hosted model; analytics and Sentry telemetry are opt out.
Alpha churn. Around 13 commits a day and PR numbers near 6,900 mean APIs move under you.
Technical Moats
Integration breadth. 60+ advertised tools across observability, cloud, databases, data platforms and incident management, each with config verification, credential handling and evidence mapping. Reproducing that is quarters of work.
Enforced architecture. Import contracts in CI, characterization tests before refactors, and an explicit rule that
SKILL.mdfiles and the system prompt are human owned, even when coding agents contribute.Operational plumbing. Approval brokers for Slack, Discord and Telegram, a write ahead log for turns, session cost tracking, AMI based gateway deploys with rollback.
Community velocity. Roughly 12,400 stars and a steady stream of contributor PRs. If the RL environment ships, this community is its scenario authoring pipeline.
Contrarian Insights
Insight One: the benchmark measures the bill, not the diagnosis. The repo's benchmark doc lists three synthetic scenarios (replication lag, connection exhaustion, storage full) and reports duration, token usage and estimated cost. It states that accuracy, false positives and false negatives are not reported. That gap is the whole problem. On OpenRCA's 335 real incidents, the AIOps 2026 study found the best of five frontier models got a perfect root cause only 12.5% of the time, and Claude Sonnet 4 just 3.9%. Until OpenSRE scores correctness, a confident, well cited answer is the only signal, and that is exactly what an LLM produces when it is wrong.
Insight Two: context management can cause the failures it prevents. The same study's top failure modes were hallucinated interpretation (71.2% of runs) and incomplete exploration (63.9%), and it found these were properties of the agent framework, not the model: prompt changes did not fix them, while giving the controller full outputs and error traces did. OpenSRE's defaults go the other way: keep head and tail, drop the middle, evict the largest result first. Those are sound defaults for a coding agent. For RCA, where the decisive line may sit in the middle of a log dump, they need to be tuned per tool, for example by preferring structured queries over raw dumps.
Surprising Takeaway
Of the 20 SKILL.md workflows shipped in the repo, 14 target GitHub and CI work (repairing CI, fixing merge conflicts and security alerts, onboarding CI) and only one targets incident investigation. Today the most refined workflows in this AI SRE framework are for build pipelines, not production outages. That is not a criticism; CI failures are cheap to reproduce and score, the very property incidents lack.
TL;DR For Engineers
OpenSRE is an Apache 2.0, public alpha framework for ReAct style SRE agents with 60+ integrations, nine LLM providers, a CLI, a Python API and chat gateways.
Every tool declares a side effect level: 186 read only, 42 mutating, 9 external by our count. The CLI gates mutating and external tools; check gateway gating yourself.
Tool output is capped at ~10,000 tokens (head + tail), and eviction drops the largest results first. Use large windows or structured queries for log heavy incidents.
The bundled benchmark reports cost and latency, not accuracy. Build your own scored incident replays before trusting automated remediation.
Research says framework structure, not prompting, drives RCA failures: full outputs and cross validation beat clever prompts.
Explain It Like I'm New
When a website slows down at 3 a.m., an on call engineer has to play detective. The clues are spread across many systems: error logs, performance charts, records of recent code changes, notes from past incidents. Finding the cause can take hours, while customers wait.
OpenSRE is a toolkit for building an AI assistant that does that detective work. It connects a language model to the systems engineers already use, lets the model decide which clues to check next, and asks it to explain the cause with links to the evidence. It can also suggest a fix, and, with permission, carry it out.
The tricky part is not connecting the tools. It is trust. An AI detective that confidently names the wrong suspect is worse than none, because people may act on it. To know how often it is right, you need practice cases with known answers, the way students need graded exams. Coding assistants improved quickly once researchers built such exams. Production outages have no good equivalent yet, because real failures are messy and hard to recreate.
OpenSRE aims to become that exam as well as the assistant. Today the assistant is well built, and the exam is still being written. How that turns out will decide whether AI can be trusted to keep systems running.
See It In Action
OpenSRE Quickstart, Tracer. opensre.com/docs/getting-started/quickstart. Install, sign in, and run a first investigation in the interactive shell.
Headless CLI guide, Tracer. opensre.com/docs/guides/headless-cli. How
opensre askruns in scripts and CI, with JSON output and tool approval allowlists.Data lake demo video, Tracer on GitHub. docs/DataLake.mp4. A recorded walkthrough shipped in the repository's docs.
OpenRCA failure taxonomy paper, Kim et al. at the AIOps 2026 workshop. aiops26-Kim.pdf. The clearest account of how RCA agents fail across 1,675 runs.
Community Conversation
OSS Insight, Tracer-Cloud/opensre analytics. Shows roughly 12,400 stars and 1,800 forks, unusual traction for an alpha infrastructure tool.
Kim, Park, Yun, Lee (Hanyang University, OKESTRO), Why do AI agents systematically fail at cloud RCA?. Argues the fixes are structural: richer controller and executor communication, verification against raw telemetry, and preprocessing for issues such as timezone misalignment.
Microsoft Research, Exploring LLM-based agents for root cause analysis. An FSE 2024 industry paper finding a ReAct agent competitive with retrieval baselines on Microsoft incidents, plus a field study with an Azure team that emphasizes tool integration and human oversight.
Trendshift, repository listing. Tracks OpenSRE among trending GitHub repositories.
AI SRE Needs Graded Incidents Before It Needs More Integrations
OpenSRE is the most complete open foundation for AI SRE agents we have read: a disciplined architecture, 60+ integrations, fail closed CLI approvals, reversible masking, and context management borrowed from coding agents. What it does not have yet is the thing its README says it exists to build: realistic, scored incident scenarios. The research is blunt that RCA agents fail structurally, and that frameworks, not prompts, move the numbers. Use OpenSRE today as an investigation assistant with read only tools. Before letting it remediate, build the answer key.
References
Why Do AI Agents Systematically Fail at Cloud Root Cause Analysis?, Kim, Park, Yun, Lee. Twelve failure types across 1,675 OpenRCA runs with five models; structural fixes beat prompt engineering.
Exploring LLM-based Agents for Root Cause Analysis, Roy et al., Microsoft. ReAct agents on production incidents, with a field deployment.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Jimenez et al. The benchmark OpenSRE wants to replicate for operations.
ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al. The think, act, observe loop at OpenSRE's core.
OpenSRE is an Apache 2.0, public alpha framework for AI SRE agents that connects a ReAct loop to 60+ tools with side effect metadata, reversible masking and context budgeting. Its benchmark reports cost, not accuracy, and its context defaults drop the middle of long logs and the largest results first, the weak spots RCA research identifies.
Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀
Buying an AI agent is the easy part
In Front's research, 47% of leaders say their agents already handle customer interactions without a human, at least sometimes. Of those, 79% have had a complaint caused by the agent's own work.
Running Agents in Customer Work is a webinar series for the support, success, and ops leaders making the calls. Two live sessions left, both at 10 a.m. PT: Oct. 27, who's accountable when your agent gets it wrong, and Nov. 3, what your agent's resolution rate doesn't tell you.
Register. One sign-up covers every session, recordings included.