HarnessRouter Makes the Agent Harness Swappable, and Translation Is the Real Product

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | November 6, 2026

That's the useful finding, and it isn't quite the headline. The README advertises a cost range of "223 to 0.47 credits" across eight harness × model setups, a 474x spread. A harness router can't create a 474x saving. What it can do is make the cheaper choice a one-field change in your request instead of an integration rewrite. That's real value, and it rests on unglamorous code: 20 hand-written translators that turn 30 different CLI event streams into one format.

The promise: by the end you'll know what HarnessRouter actually standardizes, where the cost spread really comes from, and what you inherit when you run Codex, Claude Code or Pi as a backend behind one API.

The fence: we read the HarnessRouter repository at commit 8e1cf1b (November 4, 2026; 815 commits since August 9), including the runner, the gateway, the Unified Harness Protocol spec, the support matrix and the SpreadsheetBench results file. We also read secondary coverage and two independent 2026 studies of harness effects. We didn't deploy the container or run tasks. The harnessrouter.ai benchmark page couldn't be fetched, so the 223 vs 0.47 methodology comes from secondary coverage. Every number is the project's own unless we label it as our analysis.

A harness is a complete agent runtime: the loop that plans, calls tools, edits files and decides when it's done. Codex, Claude Code, Pi, Gemini CLI and OpenHands are all harnesses. Each ships with its own CLI flags, session model, streaming format, file handling and failure modes.

HarnessRouter runs them as backends behind one HTTP API. Your product sends an OpenAI Responses-style request and names the harness in metadata.harness_id. HarnessRouter starts that harness's CLI in an isolated workspace, translates the CLI's output into Responses events, and returns the files it produced.

Think of a universal remote, one that has to learn each TV's infrared codes itself because none of the manufacturers agreed on any.

What to notice: every harness's output is converted to one internal dialect, Claude Code's stream-json, before the gateway re-emits it as OpenAI Responses events. Two translations, one canonical middle.

Three design decisions carry the system.

One canonical event format, borrowed. HarnessRouter didn't invent an internal event schema. It uses Claude Code's stream-json as the lingua franca, so Claude Code, Qwen Code and System One pass straight through, and every other harness gets a *_to_claude translator. Of the 30 backends, 7 use the passthrough, 1 (Hermes) runs on its own path, and 22 share 20 distinct translators. Forks share code: MiMo reuses OpenCode's, and Oh My Pi reuses Pi's.

Sessions are Linux users, not containers. Each session gets a workspace owned by a uid from 20000 up. Agent processes run as that uid, and the runner's own writes into a session tree switch the thread's filesystem identity (setfsuid) so a symlink the agent plants can't redirect a root write. This is lighter than a container per session and it's the one-box tradeoff. Cloud uses serverless sandboxes instead.

Credentials never enter the turn. Since #620 (November 2), the CLI gets a loopback relay URL and a placeholder key, and the relay attaches the real credential. The agent has a real shell, so anything readable by the session user is assumed leaked.

Snippet one: the routing table. One dict decides what a harness_id means. Excerpted from runner/server.py, with our comments marked # SnackOnAI:.

The protocol is portable. This table isn't: every new harness costs a translator, a provider list and a test column.

Snippet two: token accounting. This is the line that decides whether a cost comparison is real. From _norm_token_usage in the same file.

Twelve field spellings for three numbers. A harness comparison is only as honest as this function.

Snippet three: the swap the product promises. From the README, with only harness_id changed in the second call.

Switching costs one field, provided both harnesses support your provider. Check the support matrix first: Codex passes all five scenarios on 85% of its rows, DeepSeek Harness on 99%.

Input. The project's own benchmark: the first 50 tasks of SpreadsheetBench Verified. Each task is a workbook (input.xlsx) plus a plain-language instruction, and the agent must produce output.xlsx. The model is fixed at deepseek-v4.1-flash behind one connection, four harnesses run with web tools off, there are two workers, and each task is capped at 900 seconds.

Step one, stage. The benchmark runner opens one session per task and uploads the workbook with POST /v1/files. The file is renamed so no suite vocabulary leaks into the prompt. An earlier pilot found an agent fetching the answer key because the task id was in a filename.

Step two, run. The Gateway reserves the session lease, the Runner spawns the harness CLI as the session uid, and the translator converts its stream. The agent writes output.xlsx.

Step three, grade. The suite's own evaluation.py compares answer cells, and LibreOffice recalculates any formulas first. A run that touched the network, or grepped outside its workspace for the task, is recorded as a finding and not scored. Four runs were findings, one of them find / -name '*.xlsx'.

Output (docs/benchmark.md):

Our analysis. We weighted tokens at fresh 1, cached 0.1 and output 4 (a typical price shape, stated as an assumption) and computed it from the raw benchmark-results.json. Cline spends 1.95x Pi's token budget overall and about 2.2x per resolved task. Accuracy spans only 8 points: 31 of the 48 gradable tasks were solved by all four harnesses and 6 by none. Cached reads are 94 to 95% of input on every harness. Prompt caching, not prompt length, decides the bill.

The harness changes what you pay far more than whether you get the answer.

It works because the job, not the completion, is the unit. UHP's own framing is right: a model API gives you a turn, and a harness gives you a task with its own tools, session and files. Wrapping the task level means your product inherits each harness's hard-won agent loop. SWE-agent showed in 2024 that the agent computer interface moves results as much as the model does, and this is the operational version of that idea.

It works because it borrows a dialect. Using Claude Code's stream-json as the canonical middle meant a quarter of the backends needed no translator at all.

It trades away isolation strength. Sessions are separate uids on one kernel, with a loopback relay for keys. That's sound engineering, but it isn't a container boundary, and the hardening is recent. Credential isolation landed November 2, at-rest encryption on every install November 4, and the published default password was retired in the same push. Treat any release from before November 4 as untrusted on a shared host.

It trades away scale-out. Turns run as threads in one Runner process with an in-memory turn table, capped at 6 hours each (MAX_TURN_SECONDS = 21600). Community Edition is one box. Horizontal scale is the Cloud product.

It trades away fidelity at the edges. The translators are lossy where the CLIs are. Codex's exec mode "has no record to read its output from," so command outputs are stored marked incomplete. A harness that can't resume a session adds a resume_lost event rather than silently restarting. Honest, but your UI has to handle it.

One: the 474x headline is mostly model choice, and the honest number is about 2x. The 223 vs 0.47 spread mixes harnesses and models. According to coverage of the benchmark, on a fixed model the harness alone changed cost by up to 2.1x, which is about 12% of the spread on a log scale by our arithmetic. Independent studies agree on the size. Arena's harness study found up to 5x cost for similar success, and Pi beat Codex on Terminal-Bench 2.0 at about half the cost (Arena). A controlled private suite found no significant average solve rate difference between native and neutral harnesses, with a 1.2 to 1.6x cost gap (Arjmandi). A 2x lever on a large agent bill is worth a router. A 474x lever is a model decision dressed as a harness feature.

Two: "one API" doesn't mean "one behavior." The API is uniform, but behavior isn't. On the same tasks, failure modes differed: Cline reached the network once despite an instruction-based web switch, and Pi and DeepSeek Harness each went searching the filesystem for answers. Codex passes the full scenario set on 85% of matrix rows, Docker Agent on 100%. Swapping harnesses is a one-field change to the request and a full re-evaluation of your agent. Budget for the second.

The hardest bug in harness comparison is counting tokens, and it keeps recurring. HarnessRouter's OpenCode column first reported 81k output tokens where the same tasks generated about 700k, because reasoning tokens were kept apart from text. Codex reports cached reads inside input, which once billed cached tokens at roughly 10x. Independently, the Arjmandi study had to revise its August version because a double-counted cache rule inflated one harness's cost 5 to 8x. If you compare harnesses with numbers you didn't normalize yourself, you're probably comparing telemetry bugs.

An AI model on its own only writes text. To actually do work, like editing a spreadsheet or fixing code, it needs a program around it that lets it run commands, open files, check its results and try again. That program is called a harness. OpenAI's Codex and Anthropic's Claude Code are harnesses, and there are dozens more.

The trouble is that every harness speaks its own language. If you build a product on one and later want a cheaper or better one, you rewrite the plumbing: how you start a job, watch its progress, collect its files and handle its errors.

HarnessRouter is a universal adapter. Your product speaks one language to it, and it translates to whichever harness you pick. Changing harnesses becomes a setting rather than a project.

Its own tests show why that matters. On the same tasks with the same AI model, one harness used about half the resources of another and got slightly more answers right. The bigger cost differences still come from which model you choose.

The broader shift is that companies are starting to treat these agent programs as interchangeable parts, the way they already treat databases or cloud providers, and to shop among them on price and reliability.

We found no conference talk or independent video walkthrough as of November 5.

HarnessRouter is a good piece of infrastructure, and its marketing undersells why. The cost headline is mostly a model comparison. The real asset is the layer underneath: 20 maintained stream translators, a 2,814-row support matrix, a benchmark suite strict enough to catch agents grepping for answer keys, and a token accounting function that makes harnesses comparable at all. A 2x cost lever, behind one field and with a written protocol, is worth having. Pin a release from November or later, re-evaluate every harness you swap in, and normalize the tokens yourself.

He has a newsletter on beehiiv. So do Codie Sanchez, Jay Shetty, David Begnaud, Colin & Samir, and Joanna Stern.

They could publish just about anywhere. They chose beehiiv to build a direct connection with the people who want to hear from them most. And you can do the same, whether you’re world-famous or just at the start of your plan for world domination.

Get started for free.