In partnership with

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | September 19, 2026

The Promise

Most multi-agent demos fail for the same reason human teams fail: nobody wrote down what the handoff should look like.

MetaGPT's real contribution is not "AI employees." It is the typed artifact: a product requirements document, a file list, a mermaid class diagram, each with a schema the next agent can rely on. That idea has aged well. The rigid pipeline built around it has not, and the clearest evidence is the repository itself.

What this covers: the SOP pipeline from the ICLR 2024 oral paper, the message pool and its routing, the executable feedback loop, the paper's own cost tables, and what the current main branch actually runs. What this excludes: Data Interpreter, AFlow, SPO, and the commercial MGX product, except where they explain design drift.

Last week we tore down BMAD Method, which ships the same brief, PRD, architecture, stories chain as markdown skills with a human at every gate. MetaGPT is the research ancestor that tried to remove the human entirely. Reading them side by side tells you which half of the idea survives contact with production.

What It Actually Does

MetaGPT takes a one-line requirement such as metagpt "Create a 2048 game" and returns a repository: PRD, competitive analysis, system design, task breakdown, source files, and tests in ./workspace.

The paper's formula is Code = SOP(Team). A standard operating procedure, the kind a real software company runs, is encoded as a fixed sequence of roles. Each role emits a structured document. Each downstream role consumes it.

The community signal is enormous: 70.5k stars, 9k forks, 6,367 commits as of this writing, and an ICLR 2024 oral slot (top 1.2 percent of submissions). It is MIT licensed, pinned to Python 3.9 through 3.11, and needs Node plus pnpm because the Architect's diagrams are rendered through mermaid tooling.

The headline numbers from the paper:

Metric

ChatDev

MetaGPT w/o feedback

MetaGPT

Executability (1 to 4, human graded)

2.25

3.67

3.75

Running time (s)

762

503

541

Token usage

19,292

24,613

31,255

Total code lines

77.5

194.6

251.4

Tokens per line of code

248.9

126.5

124.3

Human revisions needed

2.5

2.25

0.83

Caption: MetaGPT spends 62 percent more tokens than ChatDev and gets 3.2x more code for it. Tokens per useful line halve. That is the efficiency argument for structure.

Plus 85.9 percent Pass@1 on HumanEval and 87.7 percent on MBPP with GPT-4. Hold that HumanEval number. It does not mean what it appears to mean (Insight One).

The Architecture, Unpacked

Caption: Focus on the middle band. There is no agent-to-agent channel. Every artifact goes into one pool, every role receives everything, and the "subscription" is a receiver-side filter on which Action produced the message.

Three decisions carry the design. Ranked:

One, handoffs are schemas, not sentences. The paper's argument is the telephone game: pass natural language through five agents and the requirement mutates at every hop. So each role's output is defined field by field through ActionNode objects with a key, an expected Python type, an instruction, and an example. The Architect cannot hand the Engineer a vibe. It hands a List[str] of files.

Two, routing is topic-based, keyed on the producing Action. A Message carries cause_by, the class of the Action that created it. A role declares _watch([WritePRD]) and ignores everything else. If you have run Kafka, this is a topic per producer type with every consumer group reading the whole log and discarding what it does not subscribe to. Simple, debuggable, and quietly O(roles × messages) in delivery work.

Three, the Engineer is the only role that touches reality. It writes code, runs its own tests, reads the failure, compares against the PRD and design in memory, and retries up to three times. Every other role is pure text generation. That asymmetry matters in Insight One.

The Code, Annotated

The handoff contract

# metagpt/actions/design_api_an.py  (abridged from current main)
from typing import List, Optional
from metagpt.actions.action_node import ActionNode

FILE_LIST = ActionNode(
    key="File list",
    expected_type=List[str],   # ← THIS is the trick: the Architect → Engineer handoff is a typed field
    instruction="Only need relative paths. Succinctly de
signate the correct entry file "
                "for your project based on the programming language: use main.js for "
                "JavaScript, main.py for Python, and so on for other languages.",
    example=["a.js", "b.py", "c.css", "d.html"],  # few-shot shape doubles as the format contract
)

# optional,because low success reproduction of class diagram in non py project.
DATA_STRUCTURES_AND_INTERFACES = ActionNode(
    key="Data structures and interfaces",
    expected_type=Optional[str],   # the paper's showcase artifact, now Optional
    instruction="Use mermaid classDiagram code syntax, including classes, method(__init__ etc.) "
                "and functions with type annotations, CLEARLY MARK the RELATIONSHIPS ...",
    example=MMC1,
)
# Output of the Architect for "Create a 2048 game": a document keyed by exactly these fields,
# which the Project Manager and Engineer parse back field by field instead of re-reading prose.

Caption: The comment above the class diagram node is the most honest line in the repository. The mermaid class diagram, the figure every MetaGPT slide deck shows, was demoted to Optional because it fails too often outside Python projects.

That is a production lesson worth stealing. A typed field that the model cannot reliably fill is worse than no field, because the downstream agent trusts it.

The subscription is a filter, not a channel

# metagpt/schema.py
class Message(BaseModel):
    cause_by: str = Field(default="", validate_default=True)                   # producing Action's class path
    send_to: set[str] = Field(default={MESSAGE_ROUTE_TO_ALL}, validate_default=True)  # ← broadcast by default

# metagpt/environment/base_env.py
def publish_message(self, message: Message, peekable: bool = True) -> bool:
    for role, addrs in self.member_addrs.items():
        if is_send_to(message, addrs):   # MESSAGE_ROUTE_TO_ALL matches every role
            role.put_message(message)    # copy lands in every role's msg_buffer
    self.history.add(message)            # global audit log, useful for replay and debugging

# metagpt/roles/role.py  (inside Role._observe)
news = self.rc.msg_buffer.pop_all()
self.rc.news = [
    n for n in news
    if (n.cause_by in self.rc.watch or self.name in n.send_to)   # ← THIS is the subscription
    and n not in old_messages                                    # dedupe against own memory
]

Caption: "Publish-subscribe" in the paper is broadcast plus receiver-side filtering. That keeps the topology flat (five subscriptions instead of ten pairwise channels for five roles) and makes the whole run replayable from env.history.

The n not in old_messages line is doing real work: without it, a role that re-reads its buffer after a crash would act on the same PRD twice. The serialization and breakpoint-recovery feature depends on this dedupe.

What main hires today

# metagpt/software_company.py  (current main, last commit January 2026)
company.hire(
    [
        TeamLeader(),       # new: a dispatcher that decides who works next
        ProductManager(),
        Architect(),
        Engineer2(),        # new: tool-using engineer, not the paper's Engineer
        # ProjectManager(), # ← the paper's ablation says this role matters. It is commented out.
        DataAnalyst(),
    ]
)
#     company.hire([QaEngineer()])   # also commented out

# metagpt/roles/di/role_zero.py
use_fixed_sop: bool = False   # ← the paper's pipeline is now opt-in

# metagpt/roles/di/team_leader.py
def publish_team_message(self, content: str, send_to: str):
    """... DONT omit any necessary info such as path, link, environment, programming language,
    framework, requirement, constraint from original content to team members because
    you are their sole info source."""                # ← a relay hub, in natural language

Caption: Before, five fixed stations watching typed artifacts. After, a hub agent that paraphrases work to spokes, with a docstring begging it not to drop details. The telephone game the paper was written to prevent is back, one hop wide.

It In Action

The paper publishes per-task telemetry for 70 SoftwareDev runs. Here is task two, end to end, with the paper's own numbers (Table 9, MetaGPT without executable feedback, GPT-4).

Input: Create a 2048 game for the web. Eight words. No framework, no layout, no scoring rules.

Step one, Product Manager. Observes the UserRequirement message. Emits a PRD: product goals, user stories, competitive analysis, a P0/P1 requirement pool, a UI draft, and an "Anything UNCLEAR" field that is almost always answered "no unclear points." Publishes with cause_by=WritePRD.

Step two, Architect. Its filter matches WritePRD. Emits the implementation approach, the file list, the mermaid class diagram, and the sequence diagram. Publishes WriteDesign.

Step three, Project Manager. Matches WriteDesign. Emits required packages, a per-file logic analysis, and the ordered task list. This is where file dependency order gets decided, which is why removing this role hurts (see the ablation below).

Step four, Engineer. Matches WriteTasks. Writes one file per task, reading the PRD, design, and prior files from memory each time.

Step five, the result.

code files:         3        lines of code:   198
doc files:          3        lines of docs:   235     ← more documentation than code
prompt tokens:      21,934   completion:      6,316   ← 77.6% of tokens are input
wall clock:         553.1 s  cost:            $1.04
executability:      3 of 4   human fix:       missing @app.route('/')

Caption: The cost checks out against GPT-4 8K list pricing ($0.03 input, $0.06 output per 1K tokens): $0.66 of reading plus $0.38 of writing. Most of the money is agents reading each other's documents.

Step six, what broke. The Flask entry route was never declared. Every artifact upstream was internally consistent. The failure happened at the one boundary no document described: how the web server exposes the game. A single human edit fixes it. Across all 70 tasks the average is 3.36 executability, $1.12, and 517 seconds per project.

Why This Design Works, And What It Trades Away

It works because it moves ambiguity resolution to the front, where it is cheap. The role ablation (Table 3) is the paper's strongest evidence:

Team

Agents

Lines

Cost

Human revisions

Executability

Engineer only

1

83

$0.915

10

1.0

+ Product Manager

2

112

$1.059

6.5

2.0

+ Architect

3

143

$1.204

4.0

2.5

+ Project Manager (no Architect)

3

205

$1.251

3.5

2.0

All four

4

191

$1.385

2.5

4.0

Caption: 51 percent more spend buys 4x fewer human revisions. The gains are not additive: either three-agent team stalls at 2.0 to 2.5, and only the full chain reaches 4.0.

That non-additivity is the architectural point. The Architect produces the interfaces, the Project Manager turns them into a dependency-ordered plan. Either alone is a document nobody can execute against.

It works a second time because the pool is an audit log. env.history holds every artifact in order, so a bad run is debuggable by reading documents, not by reconstructing a chat.

What it trades away:

Input-token tax. 81 percent of tokens across the 70-task average are prompt tokens (26,627 of 32,845). Every role re-reads upstream artifacts. Structure buys fewer tokens per line of code, but the bill is dominated by reading.

Model floor. The same SOP with DeepSeek Coder 33B scored 1.4 executability and took 1,186 seconds. With GPT-3.5 it scored 2.8 in 75 seconds. Process amplifies a capable model. It does not rescue a weak one.

No mid-run steering. The paper's own limitations section names it: users cannot easily interrupt an agent or set a checkpoint to restart from. A wrong PRD propagates faithfully through four more roles.

Waterfall by construction. Information flows one direction. The Engineer cannot tell the Architect the design is wrong. The executable feedback loop repairs code against the design, never the design itself.

Technical Moats

Nothing in the SOP is a moat. PRD, design, task list, code, test is a textbook waterfall, and every artifact schema is readable Python you can copy in an afternoon. BMAD, ChatDev, AgileCoder and a dozen internal tools have.

The artifact schemas are. Each ActionNode encodes a failure someone hit. The Optional on the class diagram, the "only need relative paths" clause, the "Refined" variants added for incremental development on existing repos. That is accumulated operational scar tissue, the same kind of moat BMAD's negative trigger clauses represent.

Distribution is the real moat. 70.5k stars, four README locales, a hosted Hugging Face Space, and a commercial product, MGX, that hit number one on Product Hunt in March 2025. The research brand (ICLR oral, then an AFlow oral in 2025) feeds the open-source funnel, which feeds the product.

Insights

Insight One: the HumanEval headline is mostly a parsing fix, and the SOP contributes about half a point on it.

The paper compares MetaGPT's 85.9 percent against GPT-4's published 67 percent. That is an 18.9-point gap. Now read Appendix C, Table 7, where the authors re-ran GPT-4 (gpt-4-0613) five times themselves:

Setting

GPT-4 HumanEval Pass@1 (5-run mean)

Raw API call

72.4%

Same call, code extracted with a regex

81.2%

Add a "respond only with Python" system prompt

80.0%

MetaGPT without executable feedback (85.9 minus the reported 4.2-point feedback gain)

81.7%

MetaGPT full

85.9%

Caption: 14.2 of the 18.9 headline points come from parsing GPT-4's output correctly. Of the remaining 4.7, the paper attributes 4.2 to the Engineer running tests. The multi-role SOP accounts for roughly 0.5 points on function-level code.

This is not an accusation. The authors publish the table, and HumanEval is the wrong benchmark for a system whose value is multi-file coordination. But it relocates the evidence. The case for SOPs rests on SoftwareDev, which means 7 hand-selected tasks in the main table, a 2-task role ablation, human grading on a 1 to 4 scale, and a dataset that, as the AgileCoder authors pointed out, was never publicly released. On the full 70 tasks, executability without feedback drops from 3.67 to 3.36. The effect is real. Its size is not pinned down.

The practical reading: if you adopt one piece of MetaGPT, adopt the executable feedback loop. It is the component with the cleanest measured gain, and it is the only one that checks work against reality instead of against another model's document.

Insight Two: the maintainers themselves moved off the fixed SOP, and the replacement reintroduces the exact failure mode the paper was written to prevent.

On current main, use_fixed_sop defaults to False. The default company is a TeamLeader dispatching to tool-using RoleZero agents whose base class allows up to 50 think-act steps per turn. ProjectManager and QaEngineer are commented out of software_company.py, even though the paper's own ablation shows Project-Manager-less teams stalling at 2.0 to 2.5 executability.

The TeamLeader hands work to spokes through publish_team_message(content, send_to), a free-text message whose docstring warns the model not to drop paths, frameworks, or constraints "because you are their sole info source." That is a relay node paraphrasing in natural language. It is the telephone game, shortened to one hop and guarded by a capital-letters instruction instead of a schema.

The generous interpretation is correct as far as it goes: GPT-4 era models needed the rails, and 2025 tool-calling models are strong enough that a fixed pipeline became a tax on simple requests. That is also BMAD's thesis, which compiles a oneshot route that skips planning entirely. What both projects kept is the typed artifact. What MetaGPT dropped is the guarantee that the artifact, not a summary of it, reaches the next agent. The MAST failure study from Berkeley found that MetaGPT's SOPs reduce specification and coordination failures relative to ChatDev while leaving verification as its weak spot. The redesign spends the strength to buy flexibility, and keeps the weakness.

Takeaway

MetaGPT's simulated software company writes more documentation than code. Across 70 tasks it averaged 240 lines of documents and 191.6 lines of code per project, and 81 percent of every token was an agent reading another agent's output. The artifacts are the product. The code is what falls out when the artifacts are good.

That reframes where to spend engineering effort. Most teams building agent pipelines tune the coding agent. MetaGPT's numbers say the leverage sits upstream, in the schema of what the planning roles must emit, and in the one loop that executes the result. Everything between those two is paid for in input tokens.

TL;DR For Engineers

  • MetaGPT turns a one-line requirement into a repo by chaining Product Manager, Architect, Project Manager, Engineer, and QA through typed ActionNode artifacts, not chat.

  • "Publish-subscribe" is broadcast to every role plus a receiver-side filter on cause_by. Flat topology, full replay from env.history.

  • 14.2 of the 18.9-point HumanEval gain over GPT-4 comes from parsing output correctly, per the paper's own Table 7. The SOP's contribution there is about half a point.

  • Adding management roles cost 51 percent more and cut human revisions 4x, on a 2-task ablation. Only the full four-role chain reached 4.0 executability.

  • Current main defaults to use_fixed_sop = False with a TeamLeader hub and the Project Manager commented out. Keep the schemas, question the org chart.

Explain It Like I'm New

Ask one person to design, build, and test a whole app in a single sitting and you usually get something that half works. Ask an AI language model to do the same thing and you get the same result, just faster.

The model is not lazy. It is juggling too much at once: what the user asked for, how the pieces fit together, which files exist, what each function should do. A small misunderstanding early on quietly spreads into everything that follows.

Human companies solved this problem long ago with process. A product manager writes down what to build. An architect decides how the parts connect. Engineers build to that plan. Each person hands the next a document with a known shape, so nobody has to guess what the last person meant.

MetaGPT copies that process for AI. It runs several copies of a language model, gives each one a single job title, and forces them to pass structured documents to each other instead of chatting freely. Picture an assembly line where every station has a checklist describing exactly what the part must look like before it moves on.

The idea matters because it changes the question from "how smart is the model" to "how well is the work organized." That second question is one engineers already know how to answer, and it is where much of the reliability in AI coding tools now comes from.

See It In Action

  • MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework, ICLR 2024 Oral, ICLR (talk page with slides). The authors' own framing of SOPs and structured communication, useful for seeing which claims they lead with and which they relegate to the appendix.

  • MetaGPT Software Company Space, DeepWisdom on Hugging Face (space). Type a requirement and watch each role's artifact appear in order, the fastest way to see how verbose the PRD stage really is.

  • How To Install MetaGPT, Build A Startup With One Prompt, Matthew Berman (video). A practical install and first run, linked from the official README; watch for how much of the output is documents versus code.

  • MultiAgent 101 tutorial and Colab, MetaGPT docs (tutorial). Build a three-role coder, tester, reviewer team in one short script and see _watch routing work end to end.

  • ICLR 2024 Best Papers and Talks: Benchmarks, Reasoning and Agents, Latent Space (episode). Places MetaGPT next to SWE-bench and OpenDevin, which is the right context for why function-level benchmarks undersell and oversell agent systems at once.

Community Conversation

  • Mert Cemri, Melissa Pan, Matei Zaharia, Ion Stoica and co-authors, UC Berkeley (paper). The MAST taxonomy annotated 1,600+ traces across seven frameworks including MetaGPT and found SOP structure cuts specification failures but not verification failures, the most rigorous outside audit MetaGPT has had.

  • AgileCoder authors (paper). Argue HumanEval and MBPP are the wrong yardstick for multi-agent development and note that neither SoftwareDev nor ChatDev's SRDD was released, which limits independent replication.

  • E2EDev benchmark authors (paper). Re-ran MetaGPT on HumanEval without benchmark-specific adaptation and got results below the paper's main table but close to its appendix, consistent with Insight One.

  • Walden Yan, Cognition (post). The strongest practitioner case against splitting work across agents: context gets fragmented at every handoff, the problem MetaGPT's typed artifacts were built to solve and its new free-text hub partly reopens.

  • Anthropic Engineering (post). A production orchestrator-worker system that reports large gains on breadth-first research and warns that tightly coupled tasks such as coding fit multi-agent designs poorly, a useful counterweight to the software-company metaphor.

  • MetaGPT team on X (launch thread). The MGX launch shows where the research went commercially, and why a dispatcher-led team replaced the fixed pipeline on main.

Keep The Documents, Fire The Org Chart

MetaGPT proved something that still holds: agents coordinating through typed artifacts beat agents coordinating through conversation, by a factor of two in tokens per useful line and a factor of four in human revisions.

It did not prove that a fixed five-role waterfall is the right shape, and its own maintainers have stopped acting as if it did. The durable engineering is small and portable: a schema per handoff, a pool that logs every artifact, and a loop that runs the code before anyone calls it done. Build those three and you have kept everything MetaGPT got right, whatever you name your agents.

References

MetaGPT chains Product Manager, Architect, Project Manager, Engineer, and QA agents through typed artifacts on a shared message pool, and its paper shows that structure halves tokens per line of code and cuts human revisions fourfold versus chat-based agents. The paper's own appendix shows most of its HumanEval gain comes from output parsing and test execution rather than the role pipeline, and its current main branch has made the fixed SOP optional in favor of a dispatcher hub. The lasting contribution is the schema per handoff and the run-the-code loop, not the org chart.

Keep Going

The habit from this issue: before you add another agent to a pipeline, write the schema of what it must hand off. If you cannot write that schema, the agent is a relay, and relays are where requirements go to mutate.

SnackOnAI runs this teardown weekly on the systems engineers actually deploy: agent frameworks, prompt architectures, serving stacks, and the papers whose appendices disagree with their abstracts. No announcements, no press release summaries. Subscribe at snackonai.com and join 10,000+ engineers reading it.

Forward this to whoever on your team is proposing a five-agent pipeline for a problem one agent with tests could solve.

Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad, it helps us keep building and delivering value 🚀

The 10 Best AI Stocks to Own in 2026

AI is moving from experiment… to essential.

Every major industry is integrating it.
Every major company is investing in it.

By late 2025, AI was already an $800B market — growing at a pace that could push it well beyond $1 trillion in the years ahead.

Cloud infrastructure is scaling fast.
AI-enabled devices are multiplying.
Automation is becoming standard.

But here’s the real question…

When trillions flow into this transformation — which stocks stand to benefit most?

Our new report reveals 10 AI stocks positioned across the backbone of this shift — from the companies powering the infrastructure… to those embedding intelligence into everyday systems.

If you want exposure to one of the defining growth trends of this decade, start here.

Download the Report Now

Recommended for you

View all
caret-right
© 2026 Snack On AI.
beehiivPowered by beehiiv