OSWorld's Hardest Baseline To Beat Is An Agent That Gives Up

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | September 03, 2026

Get revenue agents, workflows, and automations across every stage of your motion. Access customer data in real time through Attio's web app, MCP, API, and SDK.

Then Ask Attio anything about your business and get instant answers.

It's the CRM that runs the work behind every win.

Try Attio Now

The scoring design that makes OSWorld honest about infeasible tasks also hands every agent a free floor, and reading Table Five against Table Three tells you which reported results cleared it.

What this covers: the environment architecture, the config-driven setup and evaluation contract, what the observation and action spaces cost, and three findings that live in the tables rather than the abstract. What this excludes: the Windows subset and the OSWorld-Verified revision, both of which changed the picture after publication.

OSWorld is a real computer, wrapped in an API, with a grader. Sixteen authors from HKU, CMU, Salesforce Research and Waterloo, published April 2024, revised May, and it became the reference benchmark for computer-use agents almost immediately.

The construction is a virtual machine running Ubuntu with eight real applications: Chrome, VLC, Thunderbird, VS Code, GIMP, and LibreOffice Calc, Writer and Impress. An agent receives a screenshot at 1920 by 1080, an accessibility tree, or both. It emits pyautogui Python code. A per-task script inspects the resulting machine state and returns a reward in the zero to one range.

The statistics that matter:

That gap, 72.36 against 12.24, is the number everyone quoted. The more interesting one is 8.1%, and what it does to the leaderboard.

Note the ratios while you are here. 369 tasks share 134 evaluation scripts, so each script covers 2.75 tasks. And 302 initial states across 369 tasks means 67 tasks begin from a state another task also uses. Both facts matter later.

Caption: Follow the four coloured phases through one config file. Setup, post-process, extract, judge. The last line of the green phase is the design decision that quietly reshapes the leaderboard.

Three decisions carry this design, ranked by consequence.

One, the reward is execution-based and per-task. Not string matching against a reference trajectory, not an LLM judge. A Python function reads the actual machine state, and 134 of them exist. That is what lets an agent solve a task any way it likes, including ways the annotator never considered. The paper's own example: an agent asked to trim a video "using GIMP" reached for ffmpeg instead and passed, because the evaluator checks the output file rather than the route.

Two, tasks start mid-work. Files already downloaded, applications already open, windows already sized. The paper argues real assistance is needed at intermediate states, not at a fresh boot. The alternative, one snapshot per task, would cost gigabytes each, so setup is scripted instead.

Three, the action space is a real programming language. Not a click-and-type DSL. Agents emit pyautogui code, so a for-loop is a legal action. That expressiveness is the point, and the paper is explicit that prior benchmarks capping actions at clicks and typing "imposes an upper bound on agent learning capabilities."

Caption: Reconstructed from Figure Two and Table One. The setup phase exists partly to create the hooks the evaluation phase will need, which is why porting OSWorld to a new application is measured in days rather than hours.

Caption: An agent that emits FAIL unconditionally is not broken in a way the harness can detect. It is a policy, and it scores.

Caption: The best-performing observation format was also the one that made full ablations unaffordable, which is why the resolution and history studies use a tenth of the suite.

Input: the Amazon cookie task above, run against GPT-4V in the screenshot plus accessibility tree setting.

Step one, boot and setup. The Coordinator restores the chrome snapshot. The Task Manager launches Chrome with a remote debugging port open. The agent never sees this happen.

Step two, observe. Screenshot at 1920 by 1080 plus the filtered a11y tree, roughly six thousand tokens for ninety percent of observations.

Step three, act. The agent emits pyautogui code. It has fifteen steps and remembers the last three.

Step four, post-process and extract. The getter queries the debugging port for the cookie jar and pulls it to the host.

Step five, judge. is_cookie_deleted(cookie_data, {"type": "domains", "domains": [".amazon.com"]}) returns one or zero.

The numbers this produces. GPT-4V in that configuration scored 12.17% across all 369 tasks. Its category breakdown: OS 16.66%, Office 6.99%, Daily 24.50%, Professional 18.37%, Workflow 4.64%. Humans on the same categories: 75.00, 71.79, 70.51, 73.47, 73.27.

Human spread across the five categories is 4.49 points. GPT-4's is 23.56. The agent's variance is 5.2 times the human's, which says the failure is not a uniform capability deficit but something that switches on and off by application.

And of 550 sampled failures, more than 75% involved mouse click inaccuracy. The paper's phrasing is the one to remember: strong planning, weak execution. The agents wrote correct steps in their code comments and then clicked the wrong pixel.

It works because execution-based evaluation removes the single largest source of false negatives in agent benchmarks. Trajectory-matching benchmarks penalise correct alternative solutions by construction, and the paper names this as the flaw it set out to fix. Checking the machine state instead means any route that gets there counts.

It also works because the environment is genuinely general. Table Four's comparison is not marketing: WebArena shipped 5 execution-based evaluation functions, MiniWoB++ 125 for 125 toy tasks, and OSWorld 134 covering an entire operating system. Nothing before it supported cross-application tasks with intermediate initial states.

What it trades away:

Enormous authoring cost. Nine computer science students, three months, roughly 1,800 man-hours, split 650 on single-app tasks, 750 on workflows and 400 on double-checking. Another 400 collecting examples, and over 400 more across four rounds of checks. That is about 4.9 man-hours per task, and it is why the suite is 369 tasks rather than 3,690.

Evaluator reuse creates correlated failures. 134 scripts for 369 tasks. A bug in one evaluator propagates to nearly three tasks on average, and the paper is candid that further red teaming "could further reduce false positives and negatives," left to future work.

A fifteen-step cap that shapes the results. Hard tasks take humans over 180 seconds. Agents score 4.59% there. Some of that is the cap, not the capability, and the paper does not separate them.

The infeasible subset creates a scoring floor. Next section.

The human baseline is softer than it reads. 72.36% comes from CS students who had not used the software before. That is a defensible choice for a floor, but it is not expert human performance, and the number gets quoted as though it were.

There is no algorithm here to protect. The environment is a virtual machine, the action space is a public Python library, and the whole thing is open source. Anyone could build it. Almost nobody did, and the reason is the only moat OSWorld has.

The moat is per-application access engineering. Checking whether a task succeeded means interrogating an application that was never designed to be interrogated. Chrome will not hand over its cookie jar, so the harness launches it with a remote debugging port and queries that. VLC got the same treatment. VS Code needed a purpose-built extension. Thunderbird's stored account credentials required reverse engineering to decrypt. Every one of the eight applications needed its own route in, discovered by hand, and none of that work transfers to the ninth.

The second moat is the annotation itself: 1,800 man-hours across nine students, roughly 4.9 per task, plus 400 collecting candidate tasks from forums, tutorials and video courses, plus more than 400 across four rounds of cross-checking where two authors who did not write a task attempted it as agents. That last step is what separates a benchmark from a task list, and it does not parallelise or automate.

The third is the getter layer's handling of live data. Tasks whose correct answer changes over time, citation counts or blog contents, run crawlers at evaluation time rather than freezing an expected value at authoring. That keeps the suite from silently rotting, and it is the kind of maintenance decision that only shows up years later.

What is genuinely not a moat is the task count. Any team willing to spend the hours can add tasks, which is precisely the scalability the paper claims, and later work has done exactly that.

Insight One: an agent that refuses every task outscores most of the paper's own baselines.

Thirty of 369 tasks are infeasible, 8.13% of the suite. The reward function awards one when an agent "accurately predicts failure for an infeasible task." FAIL is a first-class action available on every task.

So consider the null policy: emit FAIL, always, immediately. It solves nothing. It scores 8.13%.

Now sort the twenty six baseline configurations in Table Five against that floor. Twenty one of them fall below it. Only five clear it: GPT-4 on the a11y tree at 12.24%, GPT-4o on the a11y tree at 11.36%, GPT-4V with screenshot plus a11y at 12.17%, GPT-4o in the same setting at 11.21%, and GPT-4V with Set-of-Mark at 11.77%.

Every reported result for Mixtral, Llama-3-70B, GPT-3.5, Gemini-Pro, Qwen-Max, CogAgent and Claude-3-Opus, in every input configuration, is worse than doing nothing at all.

The paper half-sees this. Table Six reports 16.67% on infeasible tasks against 13.34% on feasible ones, meaning agents do better on the impossible ones, and the text notes that some agents "tend to easily output FAIL and refuse to continue trying," producing false positives.

This is not a flaw in the benchmark's intent. Including infeasible tasks is right, and rewarding correct refusal is right. It is a flaw in how the aggregate is reported. A single overall percentage silently blends a capability score with a refusal score, and at these performance levels the refusal component dominates for most models. Any leaderboard using this number needs the feasible and infeasible subsets reported separately, and OSWorld-Verified later moved in that direction.

Insight Two: the multimodal benchmark's best result came from a model that could not see.

The top score in the paper is GPT-4, text only, reading a filtered accessibility tree: 12.24%. Not GPT-4V. Not any vision configuration.

You can test this cleanly because GPT-4o was run in all four input settings:

Adding the screenshot to the tree cost 0.15 points. Removing the tree cost more than half the score. And Set-of-Mark, the technique specifically designed to improve visual grounding by drawing numbered boxes on interface elements, was the worst configuration of the four, scoring 2.47 times below text alone.

The paper explains the SoM result rather than hiding it: desktop screenshots carry far more elements than web pages, spreadsheet cells especially, so the annotations become noise, and some tasks need pixel-level precision a bounding box cannot express.

The implication is uncomfortable for the framing. On the first benchmark built for multimodal computer agents, vision contributed roughly nothing on top of a structured text representation, and the leading vision-assistance technique was actively harmful. The paper's own defence of the screenshot-only setting is honest and forward-looking: it is the only configuration that needs no accessibility support, the tree is not available everywhere, and pure vision should generalise better eventually. Correct. It was also, in 2024, losing to a text tree by more than two to one.

Take a task subset where agents succeed 50.79% of the time. Minimise the application window before the agent starts. Success drops to 15.04%, a 70.4% relative collapse, from a change no human would notice.

The full perturbation study, on 28 tasks the agent handled relatively well:

The detail that explains it: agents could switch windows to some degree but failed to maximise a window as an intermediate step. They had no policy for establishing a workable screen before starting the actual task.

That is the production lesson hiding in a benchmark paper. Every headline number was measured on a screen someone else had already arranged correctly. Real desktops are cluttered, windows are wherever the user left them, and notifications arrive mid-task. An agent whose competence is contingent on a tidy default layout has not been measured on the thing you would deploy it into.

It also reframes the 12.24% headline. That figure is the best case on a curated screen. Perturb the layout in ways that cost humans nothing, and the same agents lose between a quarter and three quarters of what they had.

Testing whether software can operate a computer is harder than it sounds, and the difficulty is not the software. It is the grading.

The obvious approach is to record a person doing the task, then check whether the machine did the same thing. That fails immediately, because most tasks have many correct routes. Mark against one recording and you fail every alternative that worked.

OSWorld takes the other approach. Give the system a real computer, a real request, and then afterwards inspect the machine to see whether the thing actually happened. Did the file get created, with the right contents? Were those cookies really deleted? How it got there does not matter.

Sounds simple, and the cost is enormous. Someone has to write a bespoke checking program for every task, and each program needs a way to interrogate the application involved. Getting a browser to reveal its stored cookies, or an email client its account details, took genuine reverse engineering. Nine people spent three months on it.

The wider point is about what a benchmark actually is. It is a measuring instrument, and instruments have their own biases that get inherited by everything measured with them. A benchmark that rewards correct refusal also rewards giving up, and a benchmark measured on a tidy screen tells you nothing about a messy one.

Reading the tables carefully matters more than reading the headline number.

The 12.24% has expired. Agents passed it, then passed the human baseline, and quoting it now is quoting a photograph.

What has not expired is the design. Execution-based per-task evaluation, intermediate initial states, a real programming language as the action space, and a config file that makes setup and judgment the same artifact. Every serious computer-use benchmark since has copied that shape, which is the actual measure of the paper.

And the two things the tables say that the abstract does not are still true of benchmarks being published today. An aggregate that blends capability with refusal will flatter models that give up. A score measured on a tidy screen will not survive a real one.

Read the instrument before you read the reading.

The habit from this issue: whenever a benchmark rewards a null action, compute what the null policy scores and sort the leaderboard against it. Here that took one multiplication and reclassified twenty one of twenty six published results.

SnackOnAI runs this teardown weekly on the systems engineers actually deploy, agent harnesses, benchmarks, serving stacks, and the appendix tables that contradict the abstract. No announcements, no press release summaries. Subscribe at snackonai.com and join 10,000+ engineers reading it.

Forward this to whoever on your team is choosing an eval suite for an agent.

"When I think of revenue, I think of Attio." - Shreman Shrestha, Head of Business at Granola

Here's what that adds up to:

Start Your Trial