In partnership with
The pitch writes itself: a 5-second, 1344×768 character video with audio in about 72 seconds on one RTX 4090, 9x faster than MiniMax H3's native pipeline. It credits a sparse attention gate taken from a different model and grafted onto H3's reference-to-video mode.
The graft is real, and it's the clever part. The 9x mostly isn't. Going from 50 sampling steps to 4 does most of that work, through a separately published Turbo LoRA. Our FLOP count says 75% attention sparsity can make each transformer step at most about 1.45x faster at this sequence length. The repo's own table doesn't reconcile either: it implies native H3 takes 13 seconds per step and the dense Turbo run 47.5.
The promise: by the end you'll know exactly what this 558-line ComfyUI node does, why reference-to-video was considered incompatible with sparse attention, and how much of the reported speedup the sparsity can actually explain.
The fence: we read the ComfyUI-Ref2VA-VSA repository in full (5 commits, September 5 to October 17, 2026, Apache 2.0) and ComfyUI's MiniMax H3 model code and sparse attention node at master (0decb39, November 8). We also read the VSA paper and secondary coverage of MiniMax H3, FastH3 and the LightX2V Turbo LoRAs. Hugging Face model cards couldn't be fetched from this environment. We didn't run the workflow. All timings are the repo author's single-seed measurements, and the token and FLOP figures are our own arithmetic from the published architecture.
What It Actually Does
MiniMax H3, released August 3, 2026, is a 33.1B-parameter transformer that generates video and stereo audio together. Its Ref2VA mode (reference to video and audio) takes up to 9 images, 3 videos and 3 audio clips of a character and generates new footage of them.
FastVideo's FastH3 made H3 fast, but only for text-to-video. It distilled the model to 4 steps with DMD2 and trained VSA (Video Sparse Attention) gates that skip 90% of attention work. On one B200, a 15-second clip takes 47.2 s instead of 678.7 s. FastH3 says plainly that Ref2VA was not distilled.
This repo fills that gap without training anything:
Piece | Source | What it contributes |
|---|---|---|
Ref2VA backbone | Comfy-Org, | The reference-to-video model, INT8 with pruned modulation |
Step distillation | LightX2V Turbo Ref2V LoRA, 4-step v0.1 | 50 steps → 4. This is most of the speedup |
Sparse attention gate | 50 | A learned score for which video tiles matter |
This repo |
| Attaches the gates, tiles the sequence, keeps references dense, calls the sparse kernel |
Think of an organ transplant between twins. It works only because the donor (FastH3 text-to-video) and the recipient (H3 Ref2VA) share the same 50-block, 5,376-wide body, down to every matrix shape.
The Architecture, Unpacked
What to notice: sparsity applies only to video key tiles. Every query can still see every reference token, and reference queries see everything. That's the whole trick that makes Ref2VA compatible.
Three design decisions define the patch.
Segment-pure prefix tiles. VSA works on 64-row tiles. If one tile mixed the last text tokens with the first reference-image tokens, a single keep-or-drop decision would cover two modalities. The geometry builder pads each prefix segment to its own tiles, at the cost of a few wasted rows per segment.
A dense, exempt prefix. The kernel's sink arguments mark the first 26 tiles as always kept and always dense. That's 4% of the sequence, so it costs little and protects the conditioning signal. The repo's claim that it "guarantees strict character identity" goes further than the mechanism, as Contrarian Insight Two explains.
Borrowed, frozen gates. The repo trains nothing. It reads 50 matrices from FastH3, checks every shape against the Ref2VA blocks, and installs them as nn.Linear modules with no bias.
The Code, Annotated
Snippet one: the tiling map (excerpt from nodes.py, our comments marked # SnackOnAI:).
# Prefix: do not let a 64-token tile straddle text/audio/condition segments.
for seg_len in prefix_segments: # SnackOnAI: text, ref image, ref audio, target audio
left = seg_len
while left:
n = min(_TILE, left) # SnackOnAI: last tile of a segment may be short...
dst0 = tile_id * _TILE
untile[src:src+n] = torch.arange(dst0, dst0+n, device=device)
live.append(n) # ...and block_len tells the kernel how many rows are real
src += n; left -= n; tile_id += 1 # ← THIS is the trick: a new segment always starts a new tile
num_prefix_tiles = tile_id # SnackOnAI: everything before here becomes a dense "sink"
# Generated video is flattened t-major, then h, then w in ComfyUI H3.
for t0 in range(0, vt, 4): # SnackOnAI: 4x4x4 cubes = 64 tokens that are close
for h0 in range(0, vh, 4): # in time AND space, so one gate score per cube is meaningful
for w0 in range(0, vw, 4):
...
if int(torch.unique(untile).numel()) != total_rows:
raise RuntimeError("FastH3 VSA: untile map is not injective") # SnackOnAI: fail loudly, never scramble tokens
The geometry is computed once per shape and cached. Its sanity checks make a mis-tiling a crash, not a subtly broken video.
Snippet two: the sparse call and the memory trade (excerpt, same file).
# Tile BEFORE token-wise projections. qkv_proj and to_gate_compress have no bias,
# so zero pad rows stay zero ... avoiding four additional sequence-sized copies on 24 GB GPUs.
x_t = x.new_zeros((geo.padded_rows, x.shape[-1])) # SnackOnAI: 43,904 × 5,376 instead of 38,868 × 5,376
x_t[geo.untile_index] = x
q, k, v = attn.qkv_proj(x_t).split(inner, dim=-1) # SnackOnAI: costs 13% extra matmul on pad rows
gate_t = attn.to_gate_compress(x_t).view(1, sp, attn.heads, attn.head_dim)
del x_t # SnackOnAI: PR #3 (Mark Ledford) frees intermediates early;
# it released a 1.825 GiB padded activation in testing
keep_ratio = max(0.0, min(1.0, 1.0 - state.sparsity)) # SnackOnAI: 0.75 sparsity → keep 25% of video key tiles
out_t = ck.sol_attn(
q, k, v, scale=attn.head_dim ** -0.5,
sink_blocks=[0, geo.num_prefix_tiles], # ← THIS is the trick: references are never pruned as keys
sink_q=[0, geo.num_prefix_tiles], # ...and reference queries stay fully dense
topk_ratio=keep_ratio, tail=False,
block_len=geo.block_len, # SnackOnAI: real rows per tile, so padding never scores
coarse_gate=gate_t, # SnackOnAI: the transplanted FastH3 gate picks the tiles
)
The patch trades compute for memory: it projects padded rows (13% more matmul work) to avoid four sequence-sized scatter copies on a 24 GB card. The code comment above the call still says "90% sparsity → topk_ratio=0.10," the FastH3 setting, while the shipped workflow runs at 75%.
Snippet three: the transplant (excerpt).
for index, block in enumerate(blocks): # SnackOnAI: exactly 50, or the node refuses to run
weight = gates[index]
out_features, in_features = map(int, weight.shape) # SnackOnAI: (7168, 5376) for H3
hidden = int(getattr(block.attn.qkv_proj, "in_features", in_features))
inner = int(block.attn.heads) * int(block.attn.head_dim) # 56 heads × 128
if in_features != hidden or out_features != inner:
raise ValueError("FastH3 VSA gate shape does not match this Ref2VA base ...")
gate = nn.Linear(in_features, out_features, bias=False,
device=weight.device, dtype=weight.dtype)
gate.weight = nn.Parameter(weight.contiguous(), requires_grad=False) # ← THIS is the trick: no training,
setattr(block.attn, "to_gate_compress", gate) # just shape-checked reuse
Fifty matrices of 7,168 × 5,376 come to 1.93B parameters, about 3.6 GiB in bf16. The "gate file" is a 6% add-on to a 33.1B model, not a small adapter.
Snippet four: our arithmetic (reproducible from ComfyUI's H3 code: patch 1×2×2, VAE 16× spatial, latent_t = ((frames − 5) // 17) × 5 + 2, audio 40 Hz).
import math
W, H, F = 1344, 768, 124
lt = ((F - 5) // 17) * 5 + 2 # 37 latent frames
grid = (lt, H // 16 // 2, W // 16 // 2) # (37, 24, 42) after 2x2 patching
video = math.prod(grid) # 37,296 tokens
prefix = [150, 24 * 42, round(F / 24 * 40) * 2] # text (assumed), ref image 1,008, audio 414
d, inner, ffn = 5376, 56 * 128, 14336
S = sum(prefix) + video # 38,868
lin = 2 * (d*3*inner + inner*d + d*2*ffn + ffn*d) # projections + gated MLP, FLOPs per token
gate = 2 * d * inner # the transplanted gate: +10% linear work
attn = 4 * S * inner # dense QK^T and AV per token
print(f"attention share of block compute: {attn/(attn+lin):.0%}") # 59%
frac75, pad = 0.306, 1.13 # pair fraction at 75% (from tile counts), padding overhead
print(f"best case block speedup @75%: {(attn+lin)/(attn*frac75 + (lin+gate)*pad):.2f}x") # 1.45x
Even at 90% sparsity the same bound is 1.65x per step. Sparsity matters, but it can't carry a 9x headline.
It In Action
Input (workflows/ref2va_vsa_4step_rtx4090.json): one reference photo, the prompt "A young woman smiling warmly, looking directly at the camera...", 1344×768, 124 frames at 24 fps, seed 981445682258077, Turbo LoRA strength 1.0, gate sparsity 0.75, euler / simple, 4 steps, sigma shift 12 video / 3 audio.
Step one, encode. Qwen3-VL-32B in NVFP4 (14.6 GB file) encodes the prompt. The video VAE encodes the reference image to a 48 × 84 latent, which becomes 1,008 tokens after patching.
Step two, pack and tile. The sequence is about 150 text tokens (our estimate) + 1,008 reference + 414 target audio + 37,296 target video = 38,868 rows. Tiling produces 26 prefix tiles and 660 video tiles, 43,904 rows after padding.
Step three, sparse attention, 50 blocks × 4 steps. Each video query tile attends to the 26 prefix tiles plus the 165 highest-scoring of 660 video key tiles. That's 30.6% of dense attention pairs at 75% sparsity (16.7% at FastH3's 90%).
Step four, decode. The VAEs produce 124 frames and 32 kHz stereo audio.
Output (author's numbers, RTX 4090, single seed):
Pipeline | Steps | First run | Warm | Peak VRAM | Per step (warm) |
|---|---|---|---|---|---|
H3 native dense | 50 | ~650 s | ~650 s | ~22.5 GB | 13.0 s |
H3 Turbo LoRA, dense | 4 | ~210 s | ~190 s | ~21.0 GB | 47.5 s |
VDN-H3 | 8 | 213 s | ~135 s | ~18.5 GB | 16.9 s |
Ref2VA VSA (this repo) | 4 | 95.2 s | ~72 s | ~13.5 GB | 18.0 s |
where the 9x comes from (warm wall time)
native, 50 steps ████████████████████████████████████ 650 s
Turbo LoRA, 4 steps ███████████ 190 s ← step distillation: 3.4x
+ transplanted VSA ████ 72 s ← sparsity: 2.6x claimed, ≤1.45x by FLOPs
The dense Turbo row is the fair baseline for the gate. Even against it, the measured 2.6x exceeds what sparse FLOPs allow. The rest likely comes from memory: the dense run peaks at 21 GB on a 24 GB card, the sparse run at 13.5 GB. That's a plausible cause, but the repo doesn't measure it.
Why This Design Works (and what it trades away)
It works because the twins share a body. FastH3's text-to-video model and H3 Ref2VA share block count, hidden size, head layout and attention projections. A gate trained to rank video tiles in one carries over to the other well enough to produce coherent video.
It works because the prefix is small. References and audio are 4% of this sequence. Keeping them dense costs almost nothing and avoids the failure the README describes: tiles that straddle modalities and corrupt the conditioning.
It works because it fails loudly. The node refuses to run with the wrong block count, a missing gate, a mismatched shape, a non-BF16 tensor, head_dim other than 128, another attention override, or a stacked patch. For an unofficial graft, that's the right default.
It trades away validated quality. The gate was trained at 90% sparsity on text-to-video and is used here at 75% on reference-to-video. The 4-step Turbo LoRA (v0.1) was published for 544p and runs here at 768p. The evidence is two side-by-side videos from one seed and one face, with no identity-similarity score, no VBench, and no dense same-seed comparison at matched steps.
It trades away portability. It's CUDA only, BF16 only, batch 1 only, requires the comfy-kitchen sol_attn build, and can't stack with SageAttention or other attention patches.
It trades away exclusivity. ComfyUI core now ships its own nodes_sparse_attention.py with a VSA plan for H3: segment-pure prefix tiles, 4×4×4 cubes, prefix sinks, and a chunked QKV producer that avoids the padded full projection. What remains unique to this repo is loading a gate from a separate file onto a checkpoint that never had one.
Technical Moats
Honestly, almost none in code. It's 558 lines, Apache 2.0, built on public kernels. ComfyUI core's sparse attention node now covers the tiling and the prefix sinks.
The insight is the asset. Recognizing that the Ref2VA and T2V checkpoints share exact gate shapes, and that keeping the prefix dense removes the modality-straddling problem, is the non-obvious step. It took one person to see it before FastVideo distilled Ref2VA itself.
The guard rails. The injectivity check, the prefix accounting checks and the refusal to stack attention overrides are what make a transplant debuggable instead of mysterious.
Contrarian Insights
One: the headline is mostly step distillation, and the table doesn't add up. Fifty steps down to four accounts for about 3.4x of the 9x by the repo's own rows (650 s → 190 s). The remaining 2.6x from sparsity is more than 75% attention sparsity can deliver in FLOPs at 38,868 tokens. Our bound is about 1.45x per step, because dense attention is only 59% of block compute here and the patch adds 10% gate work and 13% padding. The per-step math is also inconsistent: native H3 at 13 s per step but dense Turbo at 47.5 s per step can't both describe the same setup without large fixed overheads, and if those overheads existed, the 72 s VSA run couldn't fit under them. The "2.24x faster than VDN" figure compares first runs. Warm, it's 1.88x. Something real is happening, plausibly memory and offloading, but the repo doesn't isolate it.
Two: a dense reference doesn't guarantee identity. Keeping reference tokens dense means the model can always see the face. It doesn't mean the generated video keeps it. Independent Ref2VA testing found that 4-step Turbo runs drift in pose and face as the clip goes on and recommends 20 to 25 steps for reference-heavy shots. The transplanted gate never saw reference tokens in training: it decides which video tiles each video query sees, and it learned that on text-only conditioning. "Guaranteeing strict character identity" is a claim about the mechanism, not a measured result.
Surprising Takeaway
The gate is the most portable part of a sparse attention model. FastH3's 50 gate matrices, 1.93B parameters, trained for text-to-video, run unmodified inside a different mode of the same model. That suggests learned sparsity is a property of the backbone's representations, not of the task. That's worth testing properly, because if it holds, every new mode of a sparse-trained base model inherits its speed for free.
TL;DR For Engineers
The node grafts FastH3's 50 trained VSA gate matrices onto MiniMax H3 Ref2VA and keeps text, reference and audio tokens dense. Only video key tiles are pruned (top 25% kept at 0.75).
For a 5-second 1344×768 clip: about 38.9K tokens, 96% of them video, tiled into 660 video and 26 prefix tiles with 13% padding.
Most of the 9x is the 4-step Turbo LoRA. By FLOPs, 75% sparsity can make each step at most about 1.45x faster. The rest of the measured gain is unexplained, plausibly memory.
Quality evidence is one seed and one face. Measure identity similarity on your own references before trusting 4 steps.
ComfyUI core now has native H3 VSA tiling. The unique value here is loading the gate onto a checkpoint that lacks one.
Explain It Like I'm New
AI video models work by repeatedly refining a noisy clip until it looks right. Each refinement pass compares every small patch of the video with every other patch, and for a 5-second high-resolution clip that's tens of thousands of patches, so the comparisons grow very fast.
Two tricks make this cheaper. The first is doing fewer passes. A "distilled" version of the model learns to get a good result in 4 passes instead of 50. The second is comparing smarter. A small learned component, called a gate, guesses which groups of patches matter for each other, and the model skips the rest.
Someone had already trained that gate for MiniMax's text-to-video mode. The project in this issue noticed that the character mode, where you give the model a photo of a person to animate, is built from exactly the same parts. So it took the gate from one and installed it in the other, while making sure the reference photo is always compared in full, never skipped.
The result runs on a single gaming graphics card in about a minute. Most of that speed comes from using fewer passes. The gate adds to it.
What makes this interesting for the industry is the reuse: a speed-up trained for one job carried over to another job without retraining.
See It In Action
Ref2VA VSA vs VDN-H3 side-by-side, Kablex on GitHub. Watch both clips. The same seed, prompt and reference at 1344×768. The repo's only quality evidence, so judge the face consistency yourself.
MiniMax H3 Day-0 support in ComfyUI, Comfy Org blog. Read the walkthrough. Explains the modulation pruning and INT8 "convrot" weights that make H3 fit on consumer GPUs at all.
How to use MiniMax H3 Ref2VA in ComfyUI, Digital Creative AI. Read the guide. Measured RTX 3090 timings (4 min 28 s for a 4-step 1344×768 Turbo run) and practical identity tips, including the drift warning.
We found no video walkthrough or independent benchmark of this specific node as of November 8.
Community Conversation
Mark Ledford (fishmongr), PR #3. The only outside contribution so far: freeing attention intermediates after their last use, which released a 1.825 GiB padded activation on a long reference video test. On 24 GB cards, lifetime management decides whether a run fits.
Satvik Paramkusam, Build Fast with AI, FastH3 review. Notes that FastH3's published speed is B200-only, that its VSA adapters are not generic LoRAs, and warns: "Do not assume the B200 timing applies to a 5090."
NYU Shanghai RITS, on the H3 license. The H3 Community License excludes the EU, UK, South Korea and the US, and requires separate authorization above US$20M in revenue. That applies to anything built on this node.
Peiyuan Zhang, Hao Zhang and the FastVideo authors, VSA at NeurIPS 2025. The original case for trainable sparsity: 2.53x fewer training FLOPs with no loss in diffusion loss, and 6x faster attention on Wan2.1-1.3B.
Conclusion: A Smart Graft Wrapped in an Overstated Benchmark
ComfyUI-Ref2VA-VSA does one genuinely clever thing. It shows that FastH3's trained sparse attention gates transplant cleanly onto H3's reference-to-video mode once the reference prefix is kept dense and tiled per segment. The code is careful and fails loudly. The marketing isn't careful: the 9x is mostly a 4-step LoRA, the per-step numbers don't reconcile, the gate runs outside its training conditions, and identity fidelity rests on one seed. Use it to make 4-step Ref2VA drafts fit and run faster on a 24 GB card. Measure identity and motion on your own references before you trust it, and watch ComfyUI core, which already ships most of the tiling logic.
References
Faster Video Diffusion with Trainable Sparse Attention (VSA), Zhang et al., NeurIPS 2025. Coarse tile selection plus fine attention in one trainable kernel.
Improved Distribution Matching Distillation for Fast Image Synthesis (DMD2), Yin et al. The distillation method behind FastH3's 4-step student.
Efficient Streaming Language Models with Attention Sinks, Xiao et al. Why always-kept "sink" tokens stabilize sparse attention, the idea behind the dense prefix.
Scalable Diffusion Models with Transformers (DiT), Peebles and Xie. The adaLN-modulated block structure H3 inherits.
ComfyUI-Ref2VA-VSA transplants FastH3's 50 trained sparse attention gates onto MiniMax H3 Ref2VA, keeping reference tokens dense and pruning 75% of video key tiles. On a 4090 it produces a 5-second 1344×768 clip in about 72 seconds. Most of that speedup comes from the 4-step Turbo LoRA, sparsity can explain only about 1.45x per step, and quality has been shown on a single seed.
Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀
Frontier models perform better in Attio’s agent harness.
We gave GPT-6 Astra the same job twice: analyze 100 sales calls and draft the follow-ups, once in Ask Attio and once in ChatGPT.
Ask Attio finished in 4 minutes 49 seconds. ChatGPT took 26 minutes 11 seconds, about 5.4 times as long.
The most ambitious GTM work requires more than frontier intelligence. It requires architecture that can put that intelligence to work at scale.