Autoresearch: The Engineering Behind Karpathy's Autonomous ML Experiment Loop

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | May 9, 2026

The conventional ML research workflow has a structural bottleneck that compute cannot fix. A researcher designs an experiment, runs it, waits for results, interprets them, designs the next experiment. The loop is serial. Sleep, meetings, context switching, and cognitive overhead impose a hard ceiling on how fast it turns. More GPUs do not help when the rate-limiting step is human judgment at each iteration.

Andrej Karpathy's autoresearch (43.3k stars, 6k forks, MIT, March 2026) removes the human from the experiment loop while keeping the human in control of the research direction. The distinction matters. The agent edits code, runs training, evaluates results, and decides what to keep. The human writes a Markdown file that defines what good research looks like, which directions to explore, and what constraints the agent must not violate. Then the human goes to sleep.

The result: approximately 12 experiments per hour, roughly 100 overnight on a single GPU. Each experiment trains a small language model for exactly 5 minutes, measures a single vocabulary-size-independent quality metric (val_bpb, validation bits per byte, lower is better), and either commits the improvement to a git branch or reverts the change. When you wake up, the git log is a complete record of every decision the agent made and why.

This newsletter dissects autoresearch as a systems engineering document: what the three-file contract enforces, why the immutable evaluation harness is the most important design decision in the entire system, how the program.md TSV history gives the agent cross-session memory, and what the bilevel autoresearch paper (arXiv:2603.23420) reveals about the next step.

Scope: autoresearch architecture (prepare.py, train.py, program.md), val_bpb evaluation harness design, agent loop protocol, git-based experiment tracking, multi-GPU parallel search, and adjacent research (arXiv:2603.23420, arXiv:2603.07114). Not covered: production nanochat (the 8xH100 version) beyond brief mention.

karpathy/autoresearch is a minimal autonomous ML experimentation framework built on top of nanochat (Karpathy's minimal LLM training codebase), stripped to a single-GPU, single-file training loop of approximately 630 lines. The design philosophy is extreme simplicity: no orchestration framework, no custom agent protocol, no infrastructure beyond git and a single Python script.

The complete system has exactly three files that matter:

prepare.py (immutable): Handles one-time data preparation. Downloads TinyStories training data, trains a BPE tokenizer with 8,192-token vocabulary, provides the dataloader and the evaluation function. This file is never modified by the human or the agent. It locks the evaluation metric in place so no experiment can game the benchmark.

train.py (agent's sandbox): Contains the full GPT model definition, the optimizer (Muon + AdamW, Karpathy's preferred combination), and the complete training loop. Everything is fair game for the agent: swap activation functions, restructure attention heads, change learning rate schedules, modify weight initialization, change model depth, change batch size. The only constraint: the code must run without crashing and must finish within the 5-minute wall-clock budget.

program.md (human's interface): Written in plain markdown. Contains the research strategy, the operating instructions for the agent, the history of previous experiments in TSV format, and the criteria for keeping versus discarding a result. This is the only file the human touches. Karpathy notes that program.md is already "90% AI written I ain't writing all that" even in his own workflow.

The architecture is a three-party contract: prepare.py is the immutable judge, train.py is the mutable candidate, and program.md is the research direction.

Focus on the immutable prepare.py. This is the architectural decision that makes the entire system valid. The agent can change anything in train.py, but it cannot change the evaluation harness in prepare.py. val_bpb is locked in place. Every experiment is measured against the same yardstick, regardless of architectural changes. Without this constraint, the agent could trivially "improve" val_bpb by changing what it measures.

The TSV experiment history embedded in program.md is the agent's memory. Every experiment result, including discards and crashes, is logged here. The agent reads this at the start of every session and uses it to avoid repeating failed experiments and to understand which directions are promising. Without this history, the agent would explore randomly rather than adaptively.

The multi-GPU pattern (one agent per GPU, one branch per GPU) is the emergent scaling property. Each agent independently explores the search space with a different random seed. The best improvements from each branch can be cherry-picked and merged. The git branch structure is the population tracking mechanism.

Input: A single H100 GPU, the autoresearch repo, overnight run (8 hours).

Starting state:

Agent run (8 hours, reconstructed from Karpathy's published experiment log):

The 18% keep rate is meaningful. A random agent accepting all changes would reduce val_bpb 18% of the time and increase it 82% of the time, producing net degradation. The agent's ability to compare against the previous best and revert is the mechanism that ensures the search is monotonically improving rather than random walking.

The fixed 5-minute wall-clock budget is the most important design decision in the entire system. It creates a single, unambiguous success criterion: lower val_bpb after exactly 5 minutes of training. This solves several problems simultaneously. A model that trains for longer will produce lower val_bpb regardless of architectural quality. The time budget equalizes all experiments, so a deeper model and a shallower model trained for the same time get a fair comparison. Architectural changes are not penalized for changing parameter count, because both the original and modified architectures get the same compute budget.

The val_bpb metric (bits per byte, lower is better) is vocabulary-size-independent. This matters because some architectural changes (changing tokenizer vocabulary, changing the model's internal dimensionality) would produce confounded results with a vocabulary-size-dependent metric. Karpathy chose val_bpb specifically to allow architectural experiments that change the tokenizer or model depth.

The git-based experiment tracking is the correct persistence mechanism. Each kept improvement is a git commit. The history in results.tsv (untracked) captures every run including discards. After an overnight run, the agent's git history is a complete record of its research. This is better than a log file because it is structured, atomic, and reversible.

What autoresearch trades away:

Experiment quality. The agent proposes one change at a time based on program.md and the recent history. It does not run ablations, does not identify confounded experiments, and does not distinguish between improvements that generalize and improvements that overfit to the TinyStories training distribution. The 15 kept improvements from an 83-experiment run are real improvements on the training metric, but they may not all transfer to other datasets.

Exploration breadth. A single agent proposing one change at a time explores the search space sequentially. The multi-GPU variant (one agent per GPU, one branch per GPU) addresses this partially, but it requires manual merging of improvements. The bilevel autoresearch paper (arXiv:2603.23420) is the research direction that addresses exploration quality by having an LLM optimize program.md itself.

Sample efficiency. An experienced ML researcher might propose the same 15 improvements across 30 experiments rather than 83, by using prior knowledge to avoid obviously bad directions. The agent has some prior knowledge (from its training data) but less than a domain expert. The tradeoff: the agent runs 24/7, never gets bored, and never needs to sleep.

The research community has immediately extended autoresearch to the question: if an AI agent optimizes train.py, can a second AI agent optimize program.md? This is the bilevel approach.

arXiv:2603.23420 (Bilevel Autoresearch: Meta-Autoresearching Itself) proposes exactly this: an outer agent that evaluates different program.md variants based on which versions produce the fastest val_bpb improvement rate, and an inner agent that executes experiments according to each program.md variant. The outer agent treats program.md as a hyperparameter and optimizes it using the inner agent's experimental results as a signal.

This is the natural extension of the three-file contract: once you accept that program.md defines the research strategy, the obvious next step is to optimize the research strategy itself. Karpathy explicitly anticipated this: "You can imagine comparing the research progress of different prompts, different agents, etc."

arXiv:2603.07114 (AutoResearch-RL) proposes a reinforcement learning extension where the agent receives a reward signal based on cumulative val_bpb improvement over time, not just per-experiment improvement. This changes the optimization target from "find the next improvement" to "find the fastest sequence of improvements," which is a fundamentally different research strategy.

The immutable prepare.py is both the moat and the constraint. The system works because prepare.py cannot be gamed. Every experiment is measured against the same fixed evaluation function. This is easy to understand, hard to maintain as the system scales. At production scale (8xH100, weeks of running), the question becomes: does the fixed evaluation metric remain a valid proxy for the actual research goal? Karpathy's current setup uses TinyStories val_bpb, which is a reasonable proxy for language model quality on a small corpus. For more complex research goals, the evaluation harness design becomes the primary research challenge.

The simplicity criterion in program.md. "All else being equal, simpler is better." This is the human's primary contribution to research taste beyond the initial experimental direction. It prevents the agent from accumulating architectural complexity that improves val_bpb on the training distribution through overfitting-like mechanisms. The agent will tend toward complexity (more parameters, more architectural details) without explicit pressure toward simplicity. The criterion in program.md is Occam's razor as a markdown instruction.

1,500 stars in the first days. The community adoption signal is strong, and the pattern is immediately reproducible. The barrier to entry is exactly one NVIDIA GPU, Python 3.10+, and the ability to write a program.md. The system has been forked and adapted to RTX consumer GPUs, Windows, different training corpora, and multi-agent variants within weeks of release.

Insight One: Autoresearch does not automate ML research. It automates ML experimentation. The distinction matters because the hard part of research is not running experiments. It is knowing which experiments to run and what the results mean.

Karpathy is explicit about this: "The goal is to engineer your agents to make the fastest research progress indefinitely." The agent runs experiments. The human (or a second agent in the bilevel setup) decides what kinds of experiments to run. A domain expert writes program.md in a way that avoids clearly bad directions and focuses on promising ones. A novice writes program.md in a way that has the agent explore randomly. The productivity multiplier from autoresearch is proportional to the quality of program.md, which is determined by human research taste. The bottleneck is not compute. It is your program.md.

Insight Two: The git branch as experiment tracking is a more important design decision than any single algorithmic choice, and it is the most under-discussed aspect of the architecture.

Every kept improvement is a git commit. Every discarded change is a git checkout train.py. The agent's entire research history is encoded in the git log, which is atomic, reversible, and human-readable. When Karpathy wants to understand why a particular improvement worked, he reads the git diff. When he wants to share a result, he shares a commit hash. When he wants to run a parallel experiment from a different starting point, he creates a new branch. Git is not just version control in this system. It is the primary data structure for experiment tracking. This is the "obvious in hindsight" design decision that makes the whole system inspectable and reproducible.

Karpathy wrote the fictional prologue to autoresearch in past tense, from the perspective of a future where autonomous agent research is the norm, and then published it in the present tense of March 2026:

"One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ritual of 'group meeting'. That era is long gone. Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures in the skies."

He means us. He means group meetings. The prologue is written from a future perspective about the present. He is running autoresearch on a single H100 and simultaneously running the larger version on 8xH100 and leaving both running indefinitely. His framing, "Part code, part sci-fi, and a pinch of psychosis," is not a joke. It is a precise description of the subjective experience of being the person who builds the tool that replaces the thing you used to do.

Autoresearch makes a precise engineering claim: you can separate the "what to try" (program.md, human territory) from the "how to try it" (the agent loop) and from "whether it worked" (prepare.py, immutable). This separation is not obvious, but it is the entire design. The immutable judge prevents metric gaming. The locked time budget equalizes experiments. The git branch makes every decision inspectable and reversible. The TSV in program.md gives the agent memory across sessions without a database.

What makes the system hard to dismiss is the repo's own README. Karpathy writes the fictional future in past tense: agents have taken over research, the codebase is now a self-modifying binary in its 10,205th generation, no one can tell if that's right or wrong. Then he adds: "This repo is the story of how it all began." Published March 2026. The prologue is a design document. The architecture it describes is already running.

When growth is often measured at the last click, you’re paying to compete for demand that was created somewhere else.

Reach people in the purchase planning phase before your competitors know these customers even exist.

With high-intent Pinterest signals on Performance TV you can reach audiences earlier where they watch the most.

Get an advertising advantage

Start Your Campaign