In partnership with
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 20, 2026
The promise: by the end of this issue you will understand how HarnessĀ² improves agents without touching model weights, what each point of improvement costs in extra runs, and why the "self" in self-improving deserves an asterisk.
The fence: we cover the HarnessĀ² repo, the paper (Xu, Chen, Pfister et al., Google), its Propose-Probe-Compose loop, the experience library, reported results on JobBench, LAB, and WorkBuddy-Bench, and the security implications. We skip benchmark construction details and the website demos.
What It Actually Does
A harness is everything around a model: system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. HarnessĀ² edits those eight components, task by task, while the model weights stay frozen.
It works at two levels:
Task level: a contrastive Propose-Probe-Compose step rewrites the harness for the task at hand, without any ground truth answer.
Harnessing level: an experience reflector distills what worked into a reusable library, so the next task starts smarter.
Results from a single step (k=1), means over three runs, per the paper and MarkTechPost's table:
Executor, harness | WorkBuddy avg | JobBench | LAB partial |
|---|---|---|---|
Gemini 3.5 Flash, OpenCode | 72.4 ā 77.5 | 31.5 ā 36.0 | 79.9 ā 81.9 |
Gemini 3.5 Flash, Codex | 74.5 ā 77.5 | 33.9 ā 35.3 | 71.6 ā 75.6 |
GLM-5, OpenCode | 64.2 ā 75.7 | 20.6 ā 24.2 | 59.6 ā 71.4 |
GLM-5, Codex | 64.4 ā 75.3 | 23.6 ā 24.8 | 58.1 ā 65.9 |
Community signal: 11.2k stars and 1.0k forks, Apache 2.0, though with only 3 commits visible, this is a research code drop, not a maintained product.
The honest one line summary: test time search over agent configurations, guided by behavioral evidence instead of a grader.
The Architecture, Unpacked
Caption: Focus on the probe. HarnessĀ² never asks "is this answer right?" It asks "did this edit change the behavior it was meant to change?", which works on tasks with no answer key.
Three decisions matter, ranked.
One: contrast instead of correctness. Open world tasks, like drafting a legal memo, have no verifier. HarnessĀ² runs each candidate on the same task and compares trajectories and deliverables against a reference run. The paper is explicit that these contrasts "inform subsequent composition rather than verify task correctness," because runs are noisy and share the model's biases.
Two: split every edit into invariant and domain. One candidate fixes general agent behavior (planning, tool use, context management); the other encodes domain procedure. Separating them lets the composer keep a good general fix even when the domain guess was wrong, and lets the reflector file lessons that transfer across domains.
Three: an evidence gate on memory. The reflector stores a lesson only if execution contrasts support it. Picking a winning output is not enough. That is the guardrail against a library that slowly fills with superstition.
The Code, Annotated
First, the actual workflow from the repo, with our notes on why each step exists.
# Research code: Linux + Python 3.12 + uv. The demo needs no credentials.
uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2 # dry walkthrough of the loop
# Model-backed runs go through Vertex AI.
cp .env.example .env # set Vertex project and model IDs
gcloud auth application-default login
harness2-setup lab # LAB needs Pandoc + Poppler; WorkBuddy needs Docker
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
--only-domains antitrust-competition --max-tasks 1 \
--dry-run # ā THIS is the trick: preview task selection before paying for runs
# Judging is a SEPARATE pass. Refinement never sees the grader, so the
# loop cannot optimize against it. Flags must match the refinement run exactly.
harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel
Caption: Separating refinement from judging is the experimental design in miniature. The loop improves on behavioral evidence, and the score arrives only afterwards.
Second, what each mode costs. The paper reports execution steps, not dollars, so we counted agent executions from its method description and ran the numbers.
def executions(k: int, mode: str) -> int:
"""Agent executions per task, from the paper's description (our count)."""
if mode == "parallel":
# k references + k invariant probes + k domain probes + k compositions,
# plus one selector pass over anonymized deliverables when k > 1
return 4 * k + (1 if k > 1 else 0)
# sequential: one reference, then each step adds 2 probes + 1 composition;
# each composition becomes the next step's reference
return 1 + 3 * k
for k in (1, 2, 5):
print(k, executions(k, "parallel"), executions(k, "sequential"))
# 1 4 4 ā THIS is the trick: even k=1 is 4x the runs of the base harness
# 2 9 7
# 5 21 16
# Paper: WB-Code parallel rises from 76.5 (k=1) to 80.8 (k=5)
gain, extra = 80.8 - 76.5, executions(5, "parallel") - executions(1, "parallel")
print(f"+{gain:.1f} points for {extra} extra runs = {gain/extra:.2f} points per run")
# +4.3 points for 17 extra runs = 0.25 points per run
Caption: The first step buys most of the gain for four runs. Scaling to k=5 adds 4.3 more points for 17 more runs. Plan budgets around k=1 or k=2.
Third, what a library entry looks like. The paper specifies the fields; this example is illustrative, not copied from a run.
<!-- EXPERIENCE.md (illustrative, following the paper's described fields) -->
## Situation
Legal drafting tasks requiring citations across multiple uploaded PDFs.
## Lesson
Invariant edit "extract and index all sources before drafting" (tool guidance + scripts)
cut repeated file reads and fixed missed citations. Domain edit adding a
jurisdiction checklist helped antitrust tasks only.
## Evidence
Probe trajectory: 14 fewer file reads than reference; deliverable cited 3 sources
the reference missed. Domain probe: no behavior change outside antitrust.
## Consequence
Adopt the invariant edit by default for document-heavy tasks; gate the checklist
on domain match.
Caption: Each lesson carries its own evidence. That is what lets a later task decide whether it applies, and what an auditor needs to roll it back.
It In Action
Input: one LAB legal task in the antitrust domain, executor GLM-5 under OpenCode, --k 1 --mode parallel.
Step one, reference. The base harness runs the task once. Benchmark average for this configuration: 59.6% partial pass rate.
Step two, propose. The editor reads the trajectory and writes two candidates: an invariant edit (for example, better context management for long documents) and a domain edit (antitrust specific procedure).
Step three, probe. Both candidates run on the same task. Their trajectories and deliverables are contrasted with the reference: which weakness each fixed, which it left alone, where it regressed.
Step four, compose. The composer merges the supported changes, citing specific observations, and runs the composed harness. Total: 4 executions for this task.
Step five, reflect. Lessons with supporting evidence go into the library for the next task.
Output: across the benchmark, GLM-5's LAB partial pass rate rises from 59.6% to 71.4%, +11.8 points, for 4x the executions of a single run.
Benchmark | Gemini vs GLM-5 gap before | Gap after one step |
|---|---|---|
WorkBuddy avg (OpenCode) | 8.2 points | 1.8 points |
LAB partial (OpenCode) | 20.3 points | 10.5 points |
JobBench (OpenCode) | 10.9 points | 11.8 points |
Caption: On two of three benchmarks, one harnessing step cut the gap between a weaker and a stronger model in half or better. On JobBench it did not, so test on your own work before assuming the gap closes.
Why This Design Works, and What It Trades Away
Why it works. Most agent failures are harness failures: the model had the capability, but the prompt, tools, or memory steered it wrong. Probing edits on the live task finds those failures directly. Requiring evidence for every adopted change keeps the loop from chasing noise, and the evidence gated library means the next task benefits. Efficiency improved too: on WorkBuddy-Bench, 3 of 4 setups used fewer steps per task, with Gemini under OpenCode falling from 25.4 to 21.9.
What it trades away.
Compute per task. At least 4 executions per task at k=1; 21 at k=5 parallel.
Latency. Probes and composition run before the final deliverable, so wall clock time multiplies too.
Reproducibility. The harness changes per task, so two runs of "the same agent" may not be the same agent.
Safety posture. The executor's network access is unrestricted by default, and the Codex executor runs with its sandbox disabled.
Technical Moats
The moat is not the loop; propose, test, and keep what works is an old idea. It is the contrast protocol without ground truth, which makes harness optimization usable on professional work where nobody can write a unit test for "good legal memo." Competing methods either need a training split with labels, like Meta-Harness, or skip evidence entirely. Harness Scaling, which adds compute without contrastive evidence, underperformed its own base harness in 7 of 12 benchmark configurations.
The second moat is benchmark coverage of real work: JobBench, LAB, and WorkBuddy-Bench cover professional, legal, office, web, and code tasks. Most harness optimization papers still report on coding or math, where verifiers exist.
Contrarian Insights
Insight One: "self-improving" is doing a lot of work in the headline. The model that edits the harness defaults to Claude Opus 4.8, not the executor being improved. GLM-5 and Gemini 3.5 Flash do not improve themselves; a stronger frontier model rewrites their instructions, tools, and skills. Part of the gain is a stronger model's judgment distilled into a weaker model's wrapper. That is still useful, but the fair comparison is "frontier editor plus cheap executor" against "frontier executor alone," and the paper's headline tables do not make it.
Insight Two: this is test time compute in disguise. Every task pays for at least four agent executions, plus editor and reflector calls, before you get the final deliverable. Gains from k=1 to k=5 work out to about 0.25 points per extra run. The right baseline is not the base harness at one run; it is the base harness given the same budget. The paper's Harness Scaling baseline shows that raw extra compute does not help, which is the strongest evidence that the contrastive design itself matters, but teams should still price HarnessĀ² as a 4x to 21x multiplier on agent spend.
Surprising Takeaway
One step of harness editing let GLM-5 under OpenCode reach 75.7 on WorkBuddy-Bench, beating Gemini 3.5 Flash's untouched base harness at 72.4. On that benchmark, a better wrapper was worth more than a stronger model. And the edits that made the difference were produced without a single ground truth label, only by comparing how the agent behaved before and after each change.
TL;DR For Engineers
HarnessĀ² edits 8 harness components per task with frozen weights; gains up to +11.8 points on LAB and +11.5 on WorkBuddy-Bench.
Propose-Probe-Compose needs no answer key: it compares behavior against a reference run of the same task.
Budget at least 4 executions per task at k=1; scaling to k=5 buys about 0.25 points per extra run.
The editor defaults to Claude Opus 4.8, so "self-improving" means "improved by a stronger model."
Turn on sandboxing and an egress allow-list before running; defaults leave network access open.
Explain It Like I'm New
Imagine you hire a capable new employee and hand them a thin onboarding binder. They do okay, but they make avoidable mistakes because the binder skips steps, points to the wrong tools, and never mentions the company's quirks.
You could hire someone smarter. Or you could fix the binder.
AI agents are the same. Around every AI model sits a "harness": its instructions, the tools it can use, its notes, and its rules. Most failures come from that harness, not from the model's raw ability.
Google's HarnessĀ² fixes the binder while the work is happening. It lets the agent try a task, then has a senior reviewer suggest two kinds of improvements: one general ("plan before acting") and one specific to the job ("check the jurisdiction first"). It tries both, compares what changed, and keeps only the changes that clearly helped. Lessons that proved useful get written down for next time.
The clever part is that it does this without knowing the right answer, by watching whether each change fixed the behavior it targeted.
The catch is that this means doing each task several times, which costs more. But it suggests a big shift: improving AI may increasingly mean improving the system around the model, not the model itself.
See It In Action
HarnessĀ² project site, Google Research. harness2.github.io. Visual walkthroughs of the Propose-Probe-Compose loop and examples of adopted harness edits.
HarnessĀ² repository and credential free demo, Google Research on GitHub. github.com/google-research/harness2. Run
harness2-demo --output runs/demo --k 2to see the full loop structure without any model calls.HarnessĀ²: Recursive Agent Harnessing for an Open World, arXiv. arxiv.org/html/2610.14224v1. The full method, ablations, and the parallel versus sequential recursion curves.
Google's HarnessĀ² explained, Data Science in Your Pocket on Medium. medium.com. A plain language walkthrough of improving agents without retraining.
Community Conversation
MarkTechPost, launch analysis. The fullest results table, including the comparison against Meta-Harness, ReasoningBank, and Harness Scaling, and the note that the default editor is Claude Opus 4.8.
PlayCISO, the security questions that come with it. Argues that an agent editing its own guardrails and tools becomes a moving attack surface, and that persistent lessons can be poisoned by prompt injection.
AlphaSignal, 11.8 points without touching the model. Frames the result as harness engineering overtaking model upgrades for practical gains.
The Menon Lab, recursive agent harnessing. A practitioner breakdown of the two recursion levels and how they compound across tasks.
Data Studios, recursive AI agent improvement. Covers the open source release and what teams need to run it.
The Harness Is Now a Trainable Artifact, and It Needs Change Control
HarnessĀ² makes a strong case that the wrapper around a model is as optimizable as the model itself, and that you can optimize it on real work without labels. One step lifted GLM-5 by up to 11.8 points and let it beat a stronger model's base harness on WorkBuddy-Bench. But it is not free and not fully self-directed: it costs at least four runs per task, leans on a frontier editor, and rewrites guardrails at runtime. Adopt the idea, budget for the compute, and put every adopted harness edit under the same review you would give a code change.
References
HarnessĀ²: Recursive Agent Harnessing for an Open World, Xu, Chen, CuiZhu, Dalvi Mishra, Ming, Yu, Han, Lee, Pfister, Google. The primary source.
Reflexion: Language Agents with Verbal Reinforcement Learning, Shinn et al. Verbal self-feedback stored as memory, an ancestor of the experience library.
Automated Design of Agentic Systems, Hu, Lu, Clune. Meta agent search over agent designs in code.
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, Khattab et al. Optimizing prompts and pipelines around frozen models with labeled metrics.
Voyager: An Open-Ended Embodied Agent with Large Language Models, Wang et al. A growing skill library as reusable agent experience.
HarnessĀ² improves AI agents without changing model weights by proposing, probing, and composing edits to eight harness components on each live task, using behavioral contrasts instead of ground truth, and storing evidence backed lessons for future tasks. It lifts frozen models by up to 11.8 points, at a cost of at least four agent executions per task and with a stronger frontier model as the default editor.
Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored adāit helps us keep building and delivering value š
Yes, that Arnold Schwarzenegger.
He has a newsletter on beehiiv. So do Codie Sanchez, Jay Shetty, David Begnaud, Colin & Samir, and Joanna Stern.
They could publish just about anywhere. They chose beehiiv to build a direct connection with the people who want to hear from them most. And you can do the same, whether youāre world-famous or just at the start of your plan for world domination.