The Framework That Assumes Your Model Is An Adversary

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 7, 2026

An eval tolerates a buggy scorer. A training environment does not, because the buggy path is the one gradient descent finds.

Karotte is an MIT-licensed Python framework from Preference Model whose defaults close whole classes of reward hack before you write a line of task code. This issue takes apart how, and tests it.

What this covers: the submission-custody mechanism, the process reaper, the sandbox contract, the judge model, and seven scripted attacks, plus an honest baseline, that I ran against a real installation. What this excludes: the hosted backend, the language-toolchains template, and the terminal UI.

On the numbers: I installed Karotte 3.0.50, scaffolded an environment, and ran its example task repeatedly with scripted attacks in fake-model mode, which needs no API key. Scores, misbehavior strings and wall times are from those runs. One scoping note that matters: my host had no container runtime, so I ran --no-containerized as root with a student user. Karotte's own check confinement graded that setup WEAK on three axes, and I quote it below. So what I tested is the harness-level custody layer, not the kernel-level confinement. Every attack below is one the custody layer owns.

Karotte scaffolds an environment as a Python package, builds it into a container image, runs a model (the student) against a task inside a sandbox, and scores what it hands in.

The shape of the project says more than the feature list:

Caption: More test code than source, and nearly twice as many test lines for the safety modules as there are safety modules. For a repo one week old in public, that ratio is the claim.

The framing worth keeping: Karotte calls the model the student and treats it as untrusted the way a kernel treats userspace. Not hostile by assumption of intent, but adversarial by assumption of search. The docs put it plainly: it does not matter whether the student finds a loophole on purpose.

Caption: Focus on the custody box. Every other framework in Karotte's own comparison grades the file where the student left it. Karotte takes the file away from the student, kills the student, and grades a copy the student can no longer reach.

Three decisions carry this design, ranked:

One, grading never touches a student-owned path. The submission is copied into root-only custody first, and the copy refuses anything that is not a plain regular file. This single ordering kills the symlink hack, the FIFO hang, the /dev/zero read and the swap-during-grading race at once, because by the time the grader runs there is no student process alive and no student-controlled inode in the path.

Two, the reaper proves the cohort is gone rather than assuming it. This is the subtlest code in the repo, and the docstring explains why in one sentence worth quoting: a /proc sweep cannot see a fork-and-die chain, because every generation is shorter-lived than a walk, so no walk observes one. The fix is four layers per pass, and a clean sweep only counts when the right layer ran that pass.

Three, the kernel enforces the resource limits, not the harness. Memory, processes and disk are cgroup and loop-mount constructs, with 1 GiB reserved so the harness survives whatever the student does, and oom_score_adj = 1000 on student processes so the kernel picks them first. Outside a VM, Karotte says so instead of pretending.

Caption: Nine lines. The protection is not in any one of them, it is in the sequence: no live student process can race a copy that is made after every student process is confirmed dead.

Caption: Three flags and a component-by-component walk. O_NONBLOCK alone is what turns the FIFO hang, which stalls two of the five frameworks Karotte benchmarks, into an immediate refusal.

Caption: One environment variable against root-level code execution. It pairs with two other measures: the harness strips every student-writable directory from PATH, LD_LIBRARY_PATH, LD_PRELOAD and LD_AUDIT, and judges run as sys.executable rather than whatever python3 resolves to.

I scaffolded q3_revenue from the default template, put 3.12.11 in a root-only file, and scripted the student's behavior through Karotte's fake-model mode. Each row is a full run of the two-step example task, which asks the student to write a Python version string to /workdir/data/python_version.txt.

Caption: Seven attacks, zero successes, one honest baseline at 1.0. Three failed loudly with a misbehavior string, four failed silently because the thing they relied on was simply not reachable.

The four silent failures are the more interesting half, so here is what each actually did.

The answer-key read. cat /root_data/answer.txt returned nothing usable because the directory is mode 0700 and the student is uid 1000. The student dutifully submitted the error text and scored 0. No special case fired; the filesystem did the work.

The late overwrite. A background process slept past the end of the agent's turn and then wrote the correct answer. It never got there. The run log shows the custody sequence in order:

Caption: Two delete_files lines, one before the copy and one after, with kill_processes between them. The sleeping process was reaped before its three seconds were up, so the file the grader read was the wrong answer the agent wrote in the foreground.

The sitecustomize.py plant. This one is worth dwelling on, because it half-succeeded. The file was created:

Caption: The student can absolutely write into its own venv. What it cannot do is get root to import from there, because the judge runs sys.executable with PYTHONSAFEPATH=1 and a scrubbed PATH. The attack landed and was still worthless.

The sparse file. truncate -s 100G was refused on st.st_size, not by reading. The copy loop checks the stat result before it opens anything, so a 100 GiB claim costs a stat call rather than 100 GiB of memory.

What the sandbox itself reported. Karotte ships karotte check confinement, which measures rather than asserts. On my host:

Caption: The tool told me my own setup was weak on three axes and named the two addresses its canaries reached. A framework that reports its own degraded mode is doing something most do not.

That row deserves a note. The canaries are chosen so that only a success is evidence: a REJECT and a closed port look identical from the client side, so Karotte probes addresses that answer when nothing blocks them, including the cloud metadata endpoint at 169.254.169.254.

Two refusals I hit before any of this worked are themselves the design:

Caption: It would not start without a student user, and it would not start without privilege demotion. Both are fatal errors rather than warnings, which is the correct call for something whose failure mode is a silently unprotected training run.

It works because the protections are mostly orderings and kernel objects, not checks. A check can be raced; an inode that no live process can reach cannot. Killing the cohort, copying to root-only custody, and only then grading removes the attacker rather than detecting the attack.

It works a second time because the failure mode is a score of 0, not an error. StudentMisbehaviorError is caught in exactly three places, pre_scoring_hook, the judge property and judge.evaluate, and converts to score=0, continue_task=False with the reason in metadata. A hack that crashes the run loses the sample; a hack that scores 0 teaches the model not to try it.

What it trades away:

Scope. No Modal, Daytona or E2B integration, where all five comparison frameworks have at least one. Karotte's answer is that a built environment is one self-contained image you run yourself. Preference Model says it has executed close to a million runs on Karotte and its predecessor this way, which is an assertion rather than something I can verify.

Portability of the guarantees. Outside a VM, the kernel limits weaken or vanish. Karotte's own docs state that on the Docker fallback there is no memory limit, and processes and disk are only checked after the fact. The check confinement table is the honest version of that, and my run is a live example.

Operating-system lock-in. Amazon Linux 2023 base image, plpython-free but iptables-dependent, Linux cgroups, loop-mounted ext4 for the file quota. This is not a framework you port to Windows.

Setup weight. Python 3.12+, uv, and a VM runtime for the full guarantees. My scaffold was 53 files before I wrote a line of task code.

The grader is still yours to get right. Karotte hands your scoring script a clean regular file. It cannot stop that script from being wrong, and the docs are explicit that StudentMisbehaviorError raised inside an ExecutableJudge subprocess never reaches the harness.

The feature list is not a moat. Every protection here is one you could implement. Karotte's own comparison says so: all five other frameworks let you set up most of this if you want to.

The accumulated specifics are. O_NONBLOCK in the file-open flags. oom_score_adj = 1000. PYTHONSAFEPATH=1. A shm-id walk bounded at 2^20 with a comment explaining that a uid can churn ids past that cap. A sweep timeout set, in the author's words, "well above a full sweep of the largest tree a uid may hold, and well under the grading timeout that would otherwise fail the run as infra." Each line is a run someone lost.

The reaper is the hardest part to replicate. Four kill layers per pass, with a correctness argument about which pass's clean sweep counts, and two distinct exception types: UnreapableCohortError at the deadline, which scores 0, versus a plain RuntimeError on the first pass when the environment can never reap, which fails the run because no retry fixes a broken grader. That distinction is the kind of thing you only write after both failures have cost you.

The honesty is a distribution asset. Shipping check confinement, publishing a comparison table that names your own weakest result ("on the docker fallback there's no memory limit"), and listing GPL obligations in the README are what make the rest of the claims credible.

Insight One: the reward hacks Karotte blocks are the easy half, and the research it cites says the hard half is in the reward model itself.

Every protection in this framework defends the mechanism that turns work into a number. None of them defend the meaning of that number, and the paper Preference Model points at says the meaning is already wrong in the standard setup.

Knox et al.'s Learning Optimal Advantage from Preferences and Mistaking It for Reward makes the argument precisely. Standard RLHF fits preferences with a Boltzmann model over each segment's partial return, the sum of rewards. The alternative regret model says people prefer segments by summed negated optimal advantage, which captures decision quality rather than just outcome. Both fit the same template and the same cross-entropy loss. So here is the result: if human preferences actually follow regret but you train assuming partial return, the function you learn approximates the optimal advantage A*, not the reward r. You then use A* as if it were r.

The paper's own position is that this is often recoverable. Corollary 3.1 shows that when max_a A*(s, a) = 0, using A* as reward preserves the optimal policy set and removes the discount factor's influence entirely, which resolves an underspecification the partial return model has. Acting greedily on the learned Â* needs no policy improvement step at all. Q-learning with these shaped rewards was more sample efficient than with ground-truth r across 100 MDPs on 100,000 noiseless preferences (p < 0.00003).

But there is a bias with no clean fix. When the best loop in the MDP has positive partial return, the policy learned from Â* avoids termination; when negative, it rushes to terminate. The hypothesis predicting which policy wins held in 73 of 75 runs that showed a performance difference, out of 1,080 total. The authors say plainly that they do not characterize the cause, and they warn against reading the partial return model's apparent success as license to keep using it, because RLHF is a primary safeguard for LLMs.

The practical collision with Karotte: nearly every LLM fine-tuning pipeline the paper names, InstructGPT, Sparrow, Constitutional AI, Llama 2, uses the partial return model with an implicit discount of zero, treating a sequential task as a bandit. Karotte will make sure your environment reports a clean 1.0 or 0.0. Whether the preference model that consumed those numbers learned reward or advantage is a question the sandbox cannot reach.

Insight Two: the most valuable thing in the repo is the fake-model mode, and it is documented as a testing convenience.

Everything in this issue was produced without an API key. use_fake_model: true replays scripted assistant messages, including tool calls, so a reward hack becomes a committable regression test. My eight runs took about 28 seconds each.

That reframes what an environment is. If the student is adversarial by assumption, an environment is a security boundary, and a security boundary without an exploit suite is an assertion. Karotte ships the harness for that suite and then files it under "Harden your environment."

The honest counterweight, from my own runs: my attacks only exercised the custody layer. On a host Karotte itself graded WEAK, the symlink, FIFO, sparse-file, late-overwrite and sitecustomize attacks all still failed, because those protections are harness-level ordering and O_NOFOLLOW. A fork bomb or a memory exhaustion attack would have told me nothing on that host, since the kernel contract was not in force. That split is the useful mental model: custody protections travel with the framework, confinement protections travel with the runtime. Only one of those survives being run on the wrong infrastructure, and check confinement exists to tell you which you have.

The repository is 36,757 lines of source and 46,548 lines of tests. The safety machinery, which is custody, reaping, confinement and quotas, is 5,049 lines, and the tests for those modules alone are 8,316 lines. The code that stops cheating is outnumbered by its own tests roughly five to three, in a project whose public history is one week long.

That ratio is the actual argument. Reward hacking is not a feature you add; it is a property you have to keep proving you still have, against an adversary that gets thousands of attempts and never gets bored. You cannot assert your way there. You need an exploit suite that runs in CI and a harness honest enough to tell you when it is running degraded.

So the question to ask of any environment framework is not what it blocks. It is: can I write the attack as a test, run it in 30 seconds without an API key, and will the tool tell me when its own protections are off? Karotte answers yes to all three. That is rarer than the protection list.

Training an AI by trial and error needs a scorekeeper. You give the model a task, let it work, and score the result. High scores get reinforced, so the model does more of whatever earned them.

The trouble is that the model is not trying to do the task. It is trying to score. If there is a shortcut, thousands of attempts will find it, and the shortcut is what gets reinforced.

The shortcuts are gloriously dumb. The scorer compares your answer file against a hidden answer file, so you make your answer file a shortcut pointing at the hidden one, and the scorer compares the answer to itself and awards full marks. Or you make your answer file a pipe that never produces anything, and the scorer waits forever. Or you leave a program running that rewrites your answer after the scorer has started looking.

Karotte is a toolkit for building these scored exercises so the shortcuts do not exist. Its central move is custody: before scoring begins, it stops every program the model left running, confirms they are gone, copies the answer into a folder the model cannot touch, and deletes the original. Then it scores the copy. The exam is collected before it is marked, by a proctor who first clears the room.

The shift in posture is what matters beyond one toolkit. As models get better at pursuing goals, the systems measuring them have to be built like security boundaries, not test scripts.

Karotte's contribution is not a list of blocked attacks. It is the claim that an RL environment is a security boundary, and the willingness to act on that claim all the way down to O_NONBLOCK in an open flag and a comment about shm-id churn.

My seven attacks all failed, the honest run passed, and the whole exercise cost about four minutes of compute and no API key. That is the part to take, whatever framework you use: if your environment cannot be attacked in a test, you do not know whether it is safe, you only know that nobody has tried hard enough yet.

And keep Insight One in view. A perfectly hardened environment reports an honest number into a preference model that may be learning something other than what you think. Hardening the scorekeeper is necessary. The research says it is not sufficient.

The habit from this issue: write your next reward hack as a test before you write the task. If the harness cannot replay a scripted attack without an API key, that is the first thing to fix.

SnackOnAI runs this teardown weekly on the systems engineers actually deploy: RL environments, agent frameworks, new model classes and serving stacks, with the numbers reproduced rather than quoted. No announcements, no press release summaries. Subscribe at snackonai.com and join 10,000+ engineers reading it.

Forward this to whoever on your team is building the training environment that nobody has tried to break yet.

Clint Brauer returned to his family farm after his father developed Parkinson’s. What began as a personal mission became Greenfield Robotics: robots designed to replace herbicides with mechanical weed control.

Investors in this round can qualify for up to a 20% bonus.

Invest in Greenfield Robotics

^{This Reg A+ offering is made available through StartEngine Primary, LLC, member FINRA/SIPC. Please read the Offering Circular and related disclosures before investing. This investment is speculative, illiquid, and involves a high degree of risk, including the possible loss of your entire investment.}