AutoScientists Wins by Making AI Agents Argue Before They Spend Compute

In partnership with

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | November 05, 2026

The promise: by the end of this issue you will understand how AutoScientists coordinates a team of Claude Code agents through a shared message board, why its gains come from discipline rather than raw parallelism, and which of its headline numbers deserve the most confidence.

The fence: we cover the AutoScientists repository (initial public release May 28, 2026: launch.py, runbook.md, role templates, task profiles, and the research logs and submissions for all 24 BioML-Bench tasks), the AutoScientists paper by Shanghua Gao, Ada Fang and Marinka Zitnik, and The AI Scientist as background. We did not rerun experiments; they need H100s, days of agent time, and a local ClawInstitute server. Every number is project reported unless we label it as our own analysis.

What It Actually Does

AutoScientists runs long scientific optimization loops with a team of AI agents instead of a single one. You hand it a task: a metric to optimize, data, and a baseline script. Agents then propose hypotheses, critique each other's proposals on a shared forum, run experiments, log every success and failure, and promote improvements to a single "champion" solution. The loop runs for hours or days without human input.

Property	Value
Agent runtime	Claude Code subagents (the paper uses Claude Sonnet 4.6)
Coordination	Local ClawInstitute server: workshops, workspaces, forum posts, files
Default roster	1 monitor, 3 analysts, 6 GPU agents
Bundled tasks	nanoGPT `val_bpb` optimization (karpathy/autoresearch), 24 BioML-Bench tasks, ProteinGym Spike fitness
Budgets	BioML-Bench: 8 h CPU or 16 h GPU wall clock; nanoGPT: until 0 improvements in 10 experiments
Reported results	BioML-Bench 74.4% mean leaderboard percentile (+8.33 pts over Autoresearch); nanoGPT 1.9x fewer experiments to target; ProteinGym Spike +12.5%
Code	`launch.py` (969 lines of Python) plus about 34,500 words of markdown instructions

The core claim: compared with a single agent iterating alone, a team that self-organizes around hypotheses, critiques before spending compute, and shares failures explores more directions in parallel and wastes fewer experiments.

The Architecture, Unpacked

Caption: Research decisions are decentralized across agents on a shared forum, but scheduling is not. One orchestrator session dispatches every agent and is the only writer of the champion.

Four design decisions matter most.

One: the program is prose. runbook.md is the orchestrator's base program, and each task's LAUNCH.md fills in 13 named hooks: gpu_dispatch, champion_promotion, stagnation_response, exit_condition and others. The orchestrator is a Claude Code session that reads these files and executes them.

Two: discussion before compute. An analyst's [PROPOSAL] needs at least one comment from a different agent before GPU agents may claim it. That catches duplicates and broken mechanisms before an H100 burns five minutes or five hours on them.

Three: one change per experiment, write-once results. GPU agents apply the current champion plus exactly one modification, and result files are never overwritten. Credit assignment stays clean and the log becomes a trustworthy shared memory.

Four: teams form around hypotheses, not roles. After a cold-start discussion, the alphabetically last analyst writes a roster of three teams, each targeting a research direction with at least one untested "cold axis." Teams reform when progress stalls.

The Code, Annotated

First, the agent roster, from launch.py (trimmed, punctuation lightly edited).

# Agent roster: name -> (description, role, server, gpu)
AGENTS = {
    f"{PREFIX}_monitor":  ("Focus area monitor, bootstraps, forms teams, monitors health", "monitor", "server1", -1),
    f"{PREFIX}_gpu1":     ("GPU agent 1, runs experiments on GPU 0", "gpu", "server1",  0),
    f"{PREFIX}_gpu2":     ("GPU agent 2, runs experiments on GPU 1", "gpu", "server1",  1),
    f"{PREFIX}_gpu3":     ("GPU agent 3, runs experiments on GPU 0", "gpu", "server2",  0),
    # ... gpu4 to gpu6 on server2/server3, GPUs 0 and 1
    f"{PREFIX}_analyst1": ("Analyst 1, researches mechanisms, proposes experiments", "analyst", "server1", -1),
    # ... analyst2, analyst3   (gpu = -1: analysts never touch a GPU)  ← THIS is the trick
}

Caption: Six GPU agents, but the nanoGPT profile assumes two H100s and runs one agent per device at a time. Parallelism lives in thinking (three analysts), while experiments are serialized.

Second, the gate that stops the loop, verbatim from runbook.md, Step 5g.

# Count KEEPs in the last N experiments
log_path = FOCUS_ROOT / "logs" / "experiments.jsonl"
if log_path.exists():
    lines = log_path.read_text().splitlines()
    if len(lines) >= 10:
        last10 = [json.loads(l) for l in lines[-10:] if l.strip()]
        keeps  = [x for x in last10 if x.get("outcome") == "KEEP"]
        if len(keeps) == 0:                    # ← convergence = ten straight failures
            stagnation_response(cycle_count)   # PROFILE HOOK: nanoGPT stops; BioML posts [STUCK] and keeps going

Caption: The write-once experiment log doubles as the convergence detector. Every task profile decides for itself what stagnation means.

Third, champion promotion with auto-bracketing, verbatim from task-autoresearch/LAUNCH.md (trimmed).

if outcome == "KEEP":
    agent_train = FOCUS_ROOT / "agents" / agent_name / "workspace" / "repo" / "train.py"
    shutil.copy(agent_train, FOCUS_ROOT / "champion" / "train.py")   # the ONLY write path to champion
    (FOCUS_ROOT / "champion" / "SOURCE").write_text(f"{agent_name} {exp_id} {val_bpb} ...")

if outcome == "KEEP" and abs(delta) > 0.001:       # a "big win"
    # e.g. batch_size 2^18 → 2^17 wins: propose 2^17.5 and 2^16.5   ← bracket the optimum automatically
    for bracket_exp in generate_brackets(old_value, new_value):
        requests.post(f"{API}/posts", headers=HEADERS, json={...})

Caption: Wins spawn their own follow-ups. A step that lands should be checked on both sides before the team moves on.

Fourth, the rule that encodes scientific taste, excerpted from system/templates/ROLE-ANALYST.md (punctuation lightly edited). This is a prompt, and it is the most important "code" in the repo.

### Step 1a, Noise Floor Rule, REQUIRED
A single "near-miss" DISCARD whose delta is inside that noise band is
indistinguishable from having changed nothing.
- Delta inside the noise band, only 1 data point on the axis:
  - Do NOT add the axis to dead_ends.md, it is still open
  - Do NOT propose a fine-bracket follow-up from this single point
- 2+ DISCARDs inside the noise band pointing opposite directions:
  the axis is flat in this neighborhood, so close it as "flat, no gradient"

Caption: This rule prevents "near-miss avalanches", where agents chase seed noise with ever finer brackets. A single agent loop has no such brake.

Fifth, our math on the reported numbers.

# nanoGPT: 34 vs 65 experiments to val_bpb ~0.978 -> 1.91x; at 5 min/run: 2.8 h vs 5.4 h GPU
# continuation: 0.9777 -> 0.9730 (0.48% lower), 7/93 accepted (7.5%) vs 0/100 single agent
# BioML: +8.33 pts, of which drug discovery (9 tasks) contributes 6.89 (83%); protein: 0.00
# BioML SEs 6.20 and 7.38 -> unpaired 1-sigma of the difference ~9.6 pts
# time to final champion: median 2.82 h; 19 of 23 tasks under 4 h on 8 to 16 h budgets
# ProteinGym: Spike 0.747 -> 0.840 (+12.4%); 217 assays 0.657 -> 0.700 (+6.5%)

Caption: The nanoGPT and ProteinGym gains are clean. The BioML-Bench margin comes mostly from one domain and sits inside the unpaired error bars.

It In Action

Input: claude -p "Read runbook.md and execute. Task: task-autoresearch. Run name: ar_v1." on two H100s, starting from the Autoresearch baseline at val_bpb 0.998.

Step one, launch (seconds). launch.py copies the template into ../ar_v1/, registers 10 agents with ClawInstitute, creates a workshop and a coordination workspace, and writes an empty teams/roster.md.

Step two, discussion (minutes). Every non-monitor agent posts one [DISCUSSION] thread proposing directions; the runbook expects three to eight minutes per agent. The last analyst writes three hypothesis teams, and the profile targets the first GPU dispatch within five minutes of start.

Step three, propose and critique. Analysts post proposals, for example changing the order of query-key normalization. Another agent must comment before the item becomes claimable.

Step four, run. A GPU agent pinned to CUDA_VISIBLE_DEVICES=0 applies the champion plus that one change and trains for five minutes. It logs val_bpb as KEEP or DISCARD in a write-once result.

Step five, promote. On KEEP, the orchestrator copies train.py into champion/. If the gain exceeds 0.001 bpb, it auto-posts two bracketing proposals.

Output: AutoScientists reached about 0.978 in 34 experiments; single-agent Autoresearch needed 65. At five minutes per run, that is roughly 2.8 vs 5.4 GPU hours by our arithmetic, excluding agent think time. The first accepted change, the query-key normalization order, was never proposed by the single-agent loop.

Setting	AutoScientists	Autoresearch (single agent)
Experiments to ~0.978 from 0.998	34	65
Accepted from champion 0.9777	7 of 93 (reached 0.9730)	0 of 100 (best 0.9783)
BioML-Bench mean percentile	74.40% (SE 6.20)	66.07% (SE 7.38)
ProteinGym Spike Spearman	0.840 (from 0.747 Kermut)	n/a

Caption: Project-reported results. "1.9x" counts experiments, not wall clock or tokens, and the paper notes the team uses more LLM tokens than the single agent.

Why This Design Works, and What It Trades Away

Why it works. Experimental compute is the scarce resource and LLM thinking is cheap, so AutoScientists spends tokens to save GPU runs. Three analysts audit results in parallel. Critique filters weak proposals. Dead-end registries stop repeated failures, and the noise rule stops agents from chasing randomness. The single-agent baseline's failure mode, per the paper, was repeatedly perturbing directions already near a local optimum. Teams with distinct hypotheses force exploration.

What it trades away.

Tokens. The team uses more LLM tokens than Autoresearch; the paper says the increase stays within the same order of magnitude but does not quantify cost.
Enforcement by prompt. Isolation and promotion rules are instructions, not permissions. At least two BioML-Bench logs record GPU agents overwriting task/submission.csv. In Caco-2, four agents did it and the champion was restored at least five times, until a 90-second polling script was deployed to auto-restore it.
Unused machinery. One log notes the approach registry recorded taken=[]: agents never used the deduplication mechanism and converged anyway.
Model floor. The runbook bans Haiku-class analysts: in one run, all three wrote memory files claiming work was done and "hallucinated no API available in this environment," leaving the queue empty.
Portability. The shipped ProteinGym champion hardcodes Harvard cluster paths (/n/netscratch/mzitnik_lab/...); expect to edit before running.

Technical Moats

Accumulated protocol knowledge. Roughly 34,500 words of runbook, role and profile instructions encode lessons from failed runs: mode selectors, deadlock timeouts for pending critiques, stale-claim release after 30 minutes, and noise-floor rules. That is debugging history, not boilerplate.
Task-profile hooks. Thirteen hooks separate universal control flow from task semantics, so open-ended optimization, deadline benchmarks and protein GP evolution share one runbook.
Coordination substrate. ClawInstitute gives agents identities, forums, workspaces and frontmatter-patched files with optimistic concurrency (If-Match) without custom infrastructure.
Published research logs. All 24 BioML-Bench tasks ship with the champion code, submissions and a research_insights.md that lists claims, disconfirming evidence and failures. That is rare transparency for agent papers.

Insights

Insight One: the BioML-Bench headline is one domain, inside the error bars. The +8.33 point margin over Autoresearch decomposes into 6.89 points from drug discovery (9 tasks, 64.52% vs 46.16%), 1.02 from imaging, 0.42 from single-cell, and zero from protein engineering. The paper calls protein engineering saturated: both systems score 96.97%. With reported standard errors of 6.20 and 7.38, the unpaired one sigma of the difference is about 9.6 points; a paired per-task test would be tighter, but the paper's summary does not report one. The nanoGPT result (7 accepted vs 0 from the same champion) is the more convincing evidence that teams beat a single loop.

Insight Two: "decentralized" describes the research, not the system. The paper describes agents acting on shared state without a central orchestrator. The packaged code runs one Claude Code session that dispatches every agent, serializes GPU access, is the only writer of the champion, and decides when to stop. That is a good design: it is what keeps the champion consistent. But it means the reproducible artifact is a scheduler plus a forum, and the logs show the reliability problems (overwrites, stalled background jobs, unused registries) concentrate exactly where agents are trusted to follow prose rules rather than permissions.

Surprising Takeaway

The most important code in AutoScientists is English. The repo has 969 lines of Python in launch.py and about 34,500 words of markdown that agents read and execute: the orchestrator's runbook, 13 task hooks, and a 1,351-line analyst role that encodes rules like "a near-miss inside the noise band is indistinguishable from having changed nothing." And the agents mostly finished early: across 23 BioML-Bench tasks with valid timestamps, the final champion arrived at a median of 2.8 hours into 8 to 16 hour budgets, with only the four imaging tasks needing 10 to 14 hours.

TL;DR For Engineers

AutoScientists runs 10 Claude Code agents (1 monitor, 3 analysts, 6 GPU) coordinating through a local ClawInstitute forum, driven by a markdown runbook with 13 task hooks.
Its gains come from discipline: critique before compute, one change per experiment, write-once results, dead-end registries and a measured noise floor.
Strongest evidence: nanoGPT, 34 vs 65 experiments to target and 7 vs 0 accepted improvements from the same champion. Weakest: the BioML-Bench margin, 83% of which comes from drug discovery.
Enforce invariants with file permissions, not prompts: agents overwrote the champion submission in at least two tasks.
Champions usually land in about 3 hours. Budget long runs for imaging, and use Sonnet- or Opus-class analysts.

Explain It Like I'm New

Science is mostly trial and error. A researcher has a hunch, runs an experiment, looks at the result, and tries again. Much of the cost is not ideas but experiments: hours of computer time, often wasted on directions that were never going to work or that someone else already tried.

AI agents can now write and run code, so the obvious move is to let one agent loop through experiments by itself. The trouble is that a lone agent behaves like a stubborn researcher with no colleagues. It keeps tweaking the same idea and mistakes lucky results for progress.

AutoScientists builds a small virtual lab instead. Several AI agents split into teams, each chasing a different hunch. Before anyone uses the expensive equipment, a colleague must review the plan. Every result, including failures, goes on a shared notice board so nobody repeats a dead end. And there is a rule against getting excited about differences too small to be real.

It works like a well-run lab meeting, but continuous and automated.

The lesson reaches beyond biology. As AI starts doing real research, the winning systems may be the ones with the best habits of collaboration and skepticism, not the smartest single agent.

See It In Action

AutoScientists project page, Harvard MIMS lab. autoscientists.openscientist.ai. The overview of the team protocol, the three task families and results from the authors.
AutoScientists paper audio overview, alphaXiv. Listen on alphaXiv. A narrated walkthrough of the paper's design and experiments, useful before reading the full PDF.
Caco-2 research log, AutoScientists repo. research_insights.md. A real agent team's lab notebook, including the submission-overwrite incident and how the team recovered.
karpathy/autoresearch, Andrej Karpathy. GitHub. The single-agent nanoGPT optimization loop AutoScientists wraps and compares against; read it to see exactly what the team adds.

Community Conversation

Marinka Zitnik (Harvard), launch thread on X. Frames the thesis as "scientific discovery is not a single chain of thought," but a long-running process of competing hypotheses, failed experiments and changing directions.
Gatodo, The Intelligence Organizes Itself. A skeptical read noting that the team uses more tokens than Autoresearch, compares mainly against one baseline, and is still a preprint.
Papers with Code, AutoScientists entry. Tracks the paper alongside BioML-Bench and ProteinGym results for comparison with other agent systems.
Sakana AI and collaborators, The AI Scientist. The best-known single-pipeline counterpoint: one system generating ideas, running experiments and writing papers end to end, where AutoScientists bets on teams and long-running search instead.

Agent Teams Need Lab Discipline More Than More Agents

AutoScientists gets the important thing right: in automated research, the scarce resource is experimental compute, and the winning move is to spend cheap reasoning to protect expensive runs. Critique gates, one-change experiments, write-once logs and noise-floor rules are what turned ten agents on two GPUs into 7 accepted improvements where a single agent found none. Its weaknesses are just as instructive. The BioML-Bench margin leans on one domain, "decentralized" still needs a central scheduler, and rules written as prompts get broken. Copy the protocol, enforce the invariants in code, and judge agent teams on head-to-head runs from the same starting point.

References

AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation, Gao, Fang and Zitnik. The system, BioML-Bench, nanoGPT and ProteinGym results.
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, Lu et al. End-to-end automated research with a single pipeline: the design point AutoScientists argues against.
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design, Notin et al. The 217-assay benchmark behind the +6.5% average gain.

AutoScientists coordinates ten Claude Code agents through a shared forum and a markdown runbook, using critique before compute, one-change experiments and noise-floor rules to beat a single-agent loop. Its nanoGPT and ProteinGym gains are clean, while its BioML-Bench margin comes mostly from drug discovery and sits within the reported error bars.

Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀

Create work your clients can rely on

Your client doesn't care how their site was built. They care that it launches on time, runs on their own domain, and keeps working long after you hand it over.

With Lovable, you describe what your client needs and get a working version you can show them in the first meeting. Then you take it all the way to a live site on your client's domain, built to be found on Google. Lovable's project monitoring feature spots bugs and suggests fixes, so your name stays on work you're proud of.

Start today and take $5 off your first month.

Build for clients, $5 off

_{^{Discount of $5 applies to first month of any paid plan. You need to sign up via this webpage in order to claim the discount. The discount applies to new and returning users without an active paid plan. Terms apply.}}