Sponsored by
A 60-second Pixar-style animated short cost $1.33. A 70-second history documentary cost $0.02. A product ad with one OpenAI API key cost $0.69. The architectural claim worth studying: there is no orchestration code. Your AI coding assistant, Claude Code, Cursor, GitHub Copilot, Codex, or Windsurf, IS the orchestrator. It reads YAML manifests and Markdown skill files, calls Python tools, and enforces quality gates before presenting the final video.
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | July 08, 2026
The standard architecture for AI workflow automation is: write code that calls agents in sequence. The code is the orchestrator; the agents are the workers. This works, and it is how most production agent systems are built. OpenMontage inverts it. There is no Python workflow runner, no LangGraph state machine, no Prefect DAG. The instruction files ARE the workflow. When you open the project in Claude Code and describe a video, Claude reads the pipeline manifest, reads the director skill for each stage, calls Python tools one by one, checkpoints state to JSON between stages, and presents creative decisions to you for approval. The agent is not being orchestrated. The agent is the orchestrator, reading structured instructions the same way a human production team reads a production brief.
This is not a demo. The AGPL-3.0 codebase includes a full quality enforcement system: pre-compose validation that blocks renders if the delivery promise is violated, post-render self-review using ffprobe and frame sampling, a slideshow risk scorer across six dimensions, and mandatory human approval gates at proposal, script, scene plan, asset review, and final publish. Every provider selection runs through a 7-dimension scoring system (task fit 30%, output quality 20%, control 15%, reliability 15%, cost efficiency 10%, latency 5%, continuity 5%) with an auditable decision log.
The research lineage that makes this readable: Voyager (arXiv:2305.16291, Wang et al., NVIDIA + CMU + Stanford + UT Austin, 2023) demonstrated that a GPT-4-driven agent equipped with a persistent skill library of JavaScript programs could master a long-horizon open-world task by building reusable capabilities over time rather than generating one-off solutions. Generative Agents (arXiv:2304.03442, Park, Liang, Bernstein et al., Stanford + Google, 2023) established observation, planning, and reflection as the three-layer architecture for agents that reason over accumulated memory rather than acting statelessly. OpenMontage implements both: the .agents/skills/ directory is Voyager's skill library applied to video production, and the Layer 2 skills/ directory is the reflection layer that synthesizes how tools should be used into reusable instruction.
Scope: OpenMontage's three-layer knowledge architecture (what exists / how to use it / how the technology works), the 12-pipeline production system, the Backlot living storyboard, the quality enforcement system, the scored provider selection, the free production path, and the documentary montage real-footage capability. Not covered: the HyperFrames SVG character animation pipeline in depth, or the full provider cost tables.
What It Actually Does
You open the OpenMontage project directory in your AI coding assistant of choice and describe what you want:
"Make a 60-second animated explainer about how neural networks learn"
The agent reads the pipeline manifest (animated_explainer), reads the stage director skill for the research stage, runs 15-25 web searches across YouTube, Reddit, Hacker News, and academic sources, produces a structured research brief with citations, then moves to proposal. At proposal: pauses. Presents you with a differentiated concept, expected cost, and tool path. You approve. The agent proceeds through script, scene plan, asset generation (with per-asset approval on a contact sheet showing takes, prompts, costs, and quality scores), edit, and composition.
Final output: a rendered video file at your specified platform profile, with WhisperX word-level subtitles burned in, audio mixed with ducking, color graded if specified. With the free tools only (Piper TTS + Archive.org + Remotion + FFmpeg), you pay nothing. With FLUX images and ElevenLabs narration, you pay a few cents. With Kling or Runway motion clips, you might pay $1-3. Budget governance blocks the execution if total spend would exceed your cap without confirmation.
Key produced demos:
Video | Tools | Cost |
|---|---|---|
"The Last Banana" (60s Pixar-style animated short) | Kling v3 via fal.ai + Google Chirp3-HD + royalty-free music + Remotion | $1.33 |
"The Library at Alexandria" (70s history elegy) | 5 custom Remotion scenes + OpenAI TTS + Pixabay music | $0.02 |
"VOID Neural Interface" (product ad) | GPT Image 1 + OpenAI TTS + royalty-free music + WhisperX | $0.69 |
"Afternoon in Candyland" (Ghibli-style anime) | 12 FLUX images + Remotion Ken Burns + particle overlays | $0.15 |
The Architecture, Unpacked
Focus on the absence of workflow code. The agent reads YAML + Markdown, calls Python functions, and checkpoints JSON. The entire orchestration logic lives in readable instruction files you can inspect and customize, not in Python control flow you have to trace through a debugger.
The Code, Annotated
Snippet One: Pipeline Manifest Structure and Agent Tool Discovery
# pipeline_defs/animated_explainer.yaml
# This is the "what exists" layer — the agent reads this first.
# Design intent: YAML is the agent's production playbook.
# No Python code runs the stages; the agent reads this and executes.
name: animated_explainer
description: AI-generated explainer with research, narration, visuals, music
version: "2.0"
# The production stages: the agent iterates through these in order
stages:
- name: research
description: Live web research to ground the video in real, current information
skill_ref: skills/pipelines/animated_explainer/research.md # Layer 2 skill
tools: # Layer 1 tools
- web_search
- youtube_analyzer
human_gate: false # ← no approval pause at research
- name: proposal
description: Differentiated concept + tool path + cost estimate
skill_ref: skills/pipelines/animated_explainer/proposal.md
tools:
- tool_registry # ← agent queries what providers are available
human_gate: true # ← PAUSES until user approves
# ← The agent presents: concept, estimated cost, tool selection rationale
# You say yes/no. Pipeline does not proceed without approval.
- name: script
skill_ref: skills/pipelines/animated_explainer/script.md
tools:
- llm_writer
human_gate: true # ← every creative decision is a pause point
- name: scene_plan
skill_ref: skills/pipelines/animated_explainer/scene_plan.md
human_gate: true
- name: assets
skill_ref: skills/pipelines/animated_explainer/assets.md
tools:
- image_generator # ← scored provider selection chooses: FLUX, Imagen, GPT Image...
- tts_provider # ← scored selection chooses: ElevenLabs, Google, OpenAI, Piper
- music_sourcer
human_gate: true # ← per-asset contact sheet: takes, prompts, cost, quality scores
- name: edit
skill_ref: skills/pipelines/animated_explainer/edit.md
tools:
- ffmpeg
- audio_mixer
- subtitle_generator
human_gate: false
- name: compose
skill_ref: skills/pipelines/animated_explainer/compose.md
tools:
- remotion_render # ← or hyperframes_render based on render_runtime
- pre_compose_validator # ← blocks if delivery promise violated
- post_render_reviewer # ← ffprobe + frame sampling after render
human_gate: true # ← you see the final video before it's "delivered"
# render_runtime is locked at PROPOSAL stage
# Swapping it silently later is a governance violation (logged in decision trail)
render_runtime: remotion # or hyperframes — decided based on visual grammar
The human_gate: true entries are the architectural invariant that keeps the system controllable. Creative tools that generate output autonomously are most useful when they pause at decisions you actually care about: the concept, the script, the visual choices. OpenMontage's gates enforce this structurally rather than hoping the agent will ask. The checkpoint protocol also means if you reject the proposal and want a different direction, the agent resumes from the proposal stage, not from scratch.
Snippet Two: Scored Provider Selection and Free Production Path
# OpenMontage: scored provider selection and free production path
# Source: tools/tool_registry.py + tools/provider_selector.py
# Design intent: every tool choice is scored, logged, and auditable
from tools.tool_registry import registry
import json
# ─── QUERY WHAT'S AVAILABLE (agent runs this at proposal time) ────────────────
registry.discover() # auto-discovers all tools/ in subdirectories
# support_envelope: what the current API key set can actually do
support_envelope = registry.support_envelope()
# ← Returns what's possible given exactly the API keys in .env
# If FAL_KEY is missing: Kling/FLUX/Veo are removed from the envelope
# If only OPENAI_API_KEY is set: "one key" production path is available
print(json.dumps(support_envelope, indent=2))
# provider_menu: ranked options for each capability
provider_menu = registry.provider_menu()
# ← Used at proposal to show the agent what choices exist for each stage
# ─── SCORED PROVIDER SELECTION ─────────────────────────────────────────────────
# Every image/video/TTS/music selection runs through a 7-dimension scorer
# The scoring system is deterministic and logged per decision
from tools.provider_selector import ProviderSelector
selector = ProviderSelector(registry)
# Agent calls this at asset generation stage
image_selection = selector.select(
task_type="image_generation",
task_context={
"style": "Pixar-style animated short",
"character_consistency": True, # ← important: Higgsfield > FLUX for this
"budget_per_image": 0.08,
}
)
# Selection result:
# {
# "provider": "fal_kling_v3",
# "score": 0.84,
# "dimensions": {
# "task_fit": 0.9, (weight 30%) — motion, character consistency ✓
# "output_quality": 0.85, (weight 20%) — Kling v3 is high quality
# "control_features": 0.8, (weight 15%) — camera motion controls ✓
# "reliability": 0.85, (weight 15%) — stable API
# "cost_efficiency": 0.7, (weight 10%) — not the cheapest option
# "latency": 0.8, (weight 5%)
# "continuity": 0.9, (weight 5%) — same provider as prior clips ✓
# },
# "alternatives_considered": ["fal_flux", "openai_gpt_image", "stable_diffusion_local"],
# "reasoning": "Kling v3 selected: character consistency requirement + motion clips needed",
# "agent_skills": ["skills/providers/kling.md"] # ← agent reads this before prompting
# }
print(f"Selected: {image_selection['provider']} (score: {image_selection['score']:.2f})")
# Output: Selected: fal_kling_v3 (score: 0.84)
# ─── THE FREE PRODUCTION PATH ──────────────────────────────────────────────────
# Zero API keys → still makes real videos
# make setup installs: Piper TTS + Archive.org support + Remotion
free_path_tools = {
"narration": "piper_tts", # ← local, offline, free
"footage": "archive_org_search", # ← Archive.org + NASA + Wikimedia Commons
"composition": "remotion_render", # ← React-based programmatic video
"post": "ffmpeg", # ← encoding, subtitles, mixing
"subtitles": "subtitle_generator", # ← auto from Piper timestamps
}
# Result: a full video with real narration, real archival footage, and proper editing
# Cost: $0.00
# For documentary montage path (real footage, no generation):
documentary_prompt = """
Make a 90-second documentary montage about what a city feels like at 4am.
Use real footage only, no narration, elegiac tone.
"""
# ← CLIP-indexed corpus built from free/open archives
# ← Agent retrieves actual motion clips semantically, edits intentionally
# ← NOT: "animate a handful of stills and call it video"
# IS: real footage pipeline with semantic retrieval and intentional editing
The alternatives_considered field in every selection log is what makes the system auditable. When the final video costs more than expected or looks different than imagined, you can trace every provider choice, see why it was selected, and understand what would have changed if a different key were available. This is not logging for debugging; it is the decision trail that makes AI-assisted production trustworthy for teams that need to explain their choices.
It In Action: $0.02 Cinematic History Elegy
Task: A 70-second cinematic meditation on the burning of the Library of Alexandria.
Step 1: Research (zero cost)
Agent runs 18 web searches:
- "Library of Alexandria size historical estimates"
- "700000 scrolls Alexandria historical consensus"
- "Caesar fire Alexandria 48 BC vs Muslim conquest 641 vs Theophilus 391"
- "emotional resonance loss of ancient knowledge"
- YouTube: existing history videos on the topic (for pacing reference)
Research brief produced:
Key facts: est. 40,000-700,000 scrolls (debated), multiple partial burns over centuries
Emotional angle: not one night, but erosion — multiple losses over 600 years
Cinematic precedent: Adam Curtis archival style, quiet horror
Step 2: Proposal (approved)
Concept: "THE LIBRARY OF ALEXANDRIA" — a 70-second history elegy
Render runtime: remotion (bespoke atelier composition mode)
Tool path: OpenAI TTS ("ash" voice) + Pixabay strings score + Remotion custom scenes
Estimated cost: $0.01-0.03
Selected approach: 5 hand-authored Remotion scenes, no shared components
Scenes proposed:
1. Illuminated manuscript page with text
2. Cascading scroll-tags animating outward
3. Burning Counter: 700,000 → 0 inside a candle's flame (Remotion counter component)
4. Charred vellum fragment with surviving Greek text
5. Empty void (negative space as conclusion)
Step 3: Asset generation (OpenAI TTS only)
TTS provider: openai_tts ("ash" voice)
Narration: ~200 words, 3 takes generated
Per-take cost: ~$0.002
Contact sheet presented: you choose take 2
Music: Pixabay strings (free, royalty-free, auto-sourced)
Energy offset: matched to slow, elegiac pacing
Scene components: all Remotion code, no image generation
BurningCounter component: React animation, Remotion spring physics
Vellum fragment: CSS filter effects, no generated image
Step 4: Compose and post-render review
Pre-compose validation: PASS
Delivery promise: "bespoke atelier, 5 scenes" ✓
Slideshow risk: 0.2 (low — all scenes are animated, not static stills) ✓
Remotion render: 70 seconds × 30fps = 2,100 frames
Render time: ~4 minutes on M3 MacBook
Output: 1920×1080, H.264, stereo audio
Post-render self-review:
ffprobe: valid H.264, 70.4s duration, stereo audio ✓
Frame sampling at 4 positions: no black frames, overlays intact ✓
Audio: RMS -18dB (acceptable), no clipping ✓
Delivery promise honored: 5 scenes, bespoke, no shared components ✓
Self-review: PASS
Total cost: $0.02 (OpenAI TTS only)
Why This Design Works, and What It Trades Away
The "agent as orchestrator, not code as orchestrator" architecture produces a codebase that is legible at multiple levels simultaneously. A human reading pipeline_defs/animated_explainer.yaml understands the production workflow without reading any Python. A human reading skills/pipelines/animated_explainer/research.md understands exactly how to execute the research stage without running the code. The Python in tools/ is the implementation of capabilities, not the implementation of logic. Logic lives in the instruction files.
This architecture is directly motivated by the agent research literature. Voyager's key insight (arXiv:2305.16291) was that a skill library of reusable programs, built incrementally as the agent encounters new challenges, produces better long-horizon performance than generating one-off solutions for each new situation. OpenMontage's Layer 3 .agents/skills/ directory implements this: knowledge about how to write effective Kling prompts, how to structure Remotion scenes, how to avoid common TTS pacing errors, accumulated in Markdown files that any compatible agent can read and reuse.
Generative Agents (arXiv:2304.03442) established the memory stream as the architecture for agents that reason over accumulated context rather than acting statelessly. OpenMontage's checkpoint protocol is the production analog: every stage writes a JSON checkpoint including the full decision log, cost snapshot, and human approval records. The agent resumes from checkpoints on interruption. The "replay run" in Backlot scrubs through the checkpointed production history the way a human reviews a retrospective.
What OpenMontage trades away:
AGPL-3.0 creates a copyleft obligation: any software that includes or links to OpenMontage code must be released under the same license. This is a meaningful constraint for commercial products that want to incorporate OpenMontage as a component without open-sourcing their own code. MIT-licensed alternatives would not have this restriction.
The system is agent-dependent. Without an AI coding assistant with a large context window (Claude Code, Cursor, Copilot, Codex, or Windsurf), the pipeline manifests and skill files are useful documentation but not executable. A team without an AI coding assistant in their workflow gets less value from the architecture.
Render times for high-quality Remotion compositions on consumer hardware can be significant. 70 seconds at 30fps = 2,100 frames. A complex composition with particles, spring physics, and audio sync might take 5-15 minutes on a MacBook M3. Cloud render infrastructure (Remotion Lambda) is supported but adds setup.
Technical Moats
The three-layer knowledge architecture at production scale. The separation of what-exists (Python tools + YAML manifests), how-to-use-it (Markdown skills), and how-external-tech-works (agent knowledge packs) is not a conceptual distinction. It is enforced by the file system and by the agent's reading pattern. When the agent calls a video generation tool, it reads the tool's skill file before writing prompts. When it selects a provider, it reads the selector's audit trail format. This architecture is the result of iterating against real production failures, not theoretical design. Reproducing it requires understanding which failure modes each layer addresses.
The scored provider selector with 7-dimension evaluation and continuity bonus. The continuity dimension (5% weight) is the one most teams would not think to include: it slightly favors using the same provider as prior clips in the same production, because switching providers mid-production introduces visual inconsistency even when individual clips look good. This insight requires having produced enough multi-clip productions to notice the failure mode. The selector also normalizes loose brief context before scoring, expanding "Pixar-style animated short with character consistency" into structured scorer-friendly signals rather than requiring a perfectly pre-shaped input.
The documentary montage pipeline with CLIP-indexed corpus retrieval. Building a semantically searchable corpus from free/open motion footage sources (Archive.org, NASA, Wikimedia Commons, Pexels, Unsplash, Pixabay) and then editing the retrieved clips into an intentional timeline is qualitatively different from "Ken Burns effect over stock images." The distinction matters commercially: a documentary about city life at 4am made from real footage at no video generation cost is a viable production mode for teams with limited budgets. Most "free AI video" systems cannot make this claim.
Insights
Insight One: The "agent is the orchestrator" architecture is not a philosophical position about software design. It is a practical choice with concrete consequences for maintainability. When the workflow logic lives in code (LangGraph state machines, Prefect DAGs, custom Python coordinators), changing the workflow requires a developer. When the workflow logic lives in YAML manifests and Markdown skills, changing the workflow requires editing a text file. OpenMontage's architecture means a non-developer who understands video production could, in principle, modify the animated_explainer pipeline manifest to add a new stage or change an approval gate. The skill files in Layer 2 could be updated by a production professional who understands filmmaking but not Python. This is the human-readable-by-design advantage that the codebase is specifically structured to provide, and it is what the agent-context files (CLAUDE.md, CURSOR.md, CODEX.md) are designed to communicate to each specific AI coding assistant.
Insight Two: The slideshow risk scorer and delivery promise enforcement are the two features that distinguish OpenMontage from AI video tools that produce disappointing output. The slideshow risk scorer runs a 6-dimension analysis before any asset is generated, catches plans that would produce "animated PowerPoint" (repetitive shots, decorative visuals with no motion, typography-over-static-background), and flags them before wasting asset generation budget. The delivery promise enforces the semantic contract of the request: if you asked for a "motion-led cinematic video," the system will not pass a composition where 80% of the frames are still images. These two features solve the most common failure mode of AI-assisted video production, not bad generation quality, but bad planning that no downstream AI can salvage.
Surprising Takeaway
OpenMontage ships with dedicated context files for five AI coding assistants: CLAUDE.md, CURSOR.md, COPILOT.md, CODEX.md, and .windsurfrules for Windsurf. All five point to a shared AGENT_GUIDE.md and PROJECT_CONTEXT.md. The existence of five separate agent context files is more interesting than it appears. It means OpenMontage treats each AI coding assistant as a different audience with different strengths and different context formats, not as interchangeable. CLAUDE.md presumably optimizes for Claude Code's specific context management and tool-use patterns. CODEX.md optimizes for Codex's agentic behavior. This is the first open-source project in this newsletter series that explicitly treats being well-understood by AI agents as a first-class product requirement, not a nice-to-have documentation task. The README even includes a section titled "If You're An OpenClaw Agent Reading This, Congratulations: You Probably Passed The Turing Test," addressing AI agent readers directly with the shortest path to becoming useful. OpenMontage is simultaneously a tool that uses AI agents and a tool optimized for AI agents to use.
TL;DR For Engineers
OpenMontage (calesthio/OpenMontage, AGPL-3.0, 34.5k stars, #1 GitHub Trending): open-source agentic video production system. 12 pipelines, 52 tools, 500+ agent skills. Works with Claude Code, Cursor, Copilot, Codex, Windsurf. No orchestration code: your AI assistant reads YAML + Markdown + calls Python.
Three-layer knowledge: Layer 1 (tools/ + pipeline_defs/ = what exists), Layer 2 (skills/ = how to use it), Layer 3 (.agents/skills/ = external technology knowledge packs). Inspired by Voyager (arXiv:2305.16291) skill library and Generative Agents (arXiv:2304.03442) memory architecture.
Every provider selection runs through a 7-dimension scorer (task fit 30%, quality 20%, control 15%, reliability 15%, cost efficiency 10%, latency 5%, continuity 5%) with auditable decision log. Quality gates: pre-compose validation, slideshow risk scoring (6 dimensions), post-render self-review (ffprobe + frame sampling), delivery promise enforcement.
Free production path: Piper TTS + Archive.org/NASA/Wikimedia + Remotion + FFmpeg = full video at $0. With API keys: $0.02-$1.33 depending on video generation providers. Budget governance: estimate → reserve → reconcile, configurable cap ($10 default).
AGPL-3.0 creates copyleft obligation for commercial integrations. Agent-dependent: requires an AI coding assistant with large context window. Render time on consumer hardware can be significant for complex compositions.
The Agent Reads the Brief. You Approve the Cuts.
OpenMontage's architecture reflects a specific theory about where human judgment belongs in AI-assisted production. Not in writing the workflow code. Not in selecting between providers. Not in managing API calls. In approving concepts, scripts, visual choices, and final output. The system is designed to make every other decision autonomously and correctly, so the human approval gates are the high-signal creative choices rather than the operational overhead.
Whether this theory is right depends on how much you trust a scored provider selector and a 6-dimension slideshow risk analyzer. The $0.02 Library of Alexandria video suggests the trust is warranted for certain production modes. The AGPL-3.0 license and 34.5k stars suggest the community agrees.
References
OpenMontage GitHub Repository, calesthio, AGPL-3.0
Voyager: An Open-Ended Embodied Agent with Large Language Models, arXiv:2305.16291, Wang et al., NVIDIA + CMU + Stanford, 2023
Generative Agents: Interactive Simulacra of Human Behavior, arXiv:2304.03442, Park, Liang, Bernstein et al., Stanford + Google, 2023
Remotion Documentation, React-based programmatic video
OpenMontage (calesthio/OpenMontage, AGPL-3.0, 34.5k stars) is the first open-source agentic video production system where the AI coding assistant (Claude Code, Cursor, Copilot, Codex, or Windsurf) is the orchestrator, reading YAML pipeline manifests and Markdown director skills rather than being orchestrated by Python workflow code. The three-layer knowledge architecture (Layer 1: Python tools + YAML manifests; Layer 2: Markdown skills; Layer 3: agent technology knowledge packs) is grounded in the Voyager skill library pattern (arXiv:2305.16291) and the Generative Agents memory architecture (arXiv:2304.03442). The system includes 12 production pipelines, 52 tools, a 7-dimension scored provider selector with auditable decision log, mandatory human approval gates at every creative decision point, pre-compose validation, a 6-dimension slideshow risk scorer, post-render self-review via ffprobe and frame sampling, and budget governance with configurable caps. Demonstrated production costs range from $0.02 (5-scene history elegy, OpenAI TTS only) to $1.33 (60-second Pixar-style animated short with Kling motion clips); full free production is possible with Piper TTS, Archive.org footage, Remotion, and FFmpeg.
Sponsored Ad
If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad — it helps us keep building and delivering value 🚀
Early to AI winners averaging +85% in last 18+ months –> Now sharing next investment insights with you
Over the past 19 months, MavSource's founder was early to the AI winners. Sandisk +196%. Micron +150%. Western Digital +129%. Nvidia +69%. Lam Research +63%. Okta +33%. Average across eight positions: +85%.
Now that same signal comes to your inbox daily. MavSource aggregates all major AI newsletters, podcasts, AI labs, earnings calls, investor letters, and GitHub — then summarizes what matters, analyzes the trends, and adds founder commentary on why it may matter. Built for investors, founders, and operators who want to see what's next before it's a headline.
One 5-minute email, every morning. Free. Results as of June 2026.
Past performance does not indicate future results. MavSource is informational only, not investment advice.