Sponsored by
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 14, 2026
The promise: by the end of this issue you will understand what Subquadratic Sparse Attention (SSA) actually buys you, why its speedups look the way they do, and the two places a 12M token context will still break in production.
The fence: we cover SubQ 1.1 Small as described in its June 16, 2026 technical report and model card, Appen's third party verification, and the comparison to DeepSeek style sparse attention and hybrid Mamba models. We skip the May preview's product claims (SubQ Code, SubQ Search), pricing (unpublished), and the company's "zero attention" roadmap, which has no public artifact.
What It Actually Does
SubQ 1.1 Small is not a model trained from scratch. Subquadratic took an unnamed open weight frontier model, ripped out its dense attention, and replaced it with SSA, a learned, content dependent sparse attention. Then it ran about one trillion tokens of long context continued pretraining, mostly at 1M tokens, on books, long documents, and repository scale code.
Context was extended in stages with YaRN (a RoPE scaling method that stretches positional encodings): 262K → 512K → 1M → 2M. The model was trained up to 2M and evaluated up to 12M.
The headline numbers from the report:
Signal | SubQ 1.1 Small | Context |
|---|---|---|
Needle in a haystack | 100% at 1M and 2M, 98% at 6M and 12M | 50 UUID needles per length, Appen verified |
RULER, 13 tasks | 99.12% | Reported only at 128K |
GPQA Diamond | 85.4% | Sonnet 4.6 87.5, GPT-5.5 93.2, Haiku 4.5 67.2 |
LiveCodeBench v6 (pass@4) | 89.7% | Opus 4.8 92.2, GPT-5.5 92 |
AutomationBench Finance | 13% | GPT-5.5 18%, Opus 4.8 16%, Sonnet 4.6 8% |
Attention FLOPs at 1M | 64.5x less than dense | 3.9 vs 252 PFLOP per layer |
Attention latency at 1M | 966 ms vs 54,164 ms | Versus FlashAttention-2, one layer, one H100 |
CTO Alex Whedon told The New Stack the model has fewer than 100B parameters. Access is gated to design partners.
Cut through it and the claim is narrow and specific: Sonnet class quality, with attention that costs O(n·k) instead of O(n²). The rest of this issue is about what that k is.
The Architecture, Unpacked
Caption: Focus on the middle box. Every token is stored and scanned by selection, but only about 16K get real attention. That fixed k is the whole design.
Three design decisions matter, ranked:
One: transplant, don't pretrain. Converting a donor model costs roughly a trillion tokens instead of the 15T+ a frontier pretrain needs. It also means GPQA and LiveCodeBench strength is partly inherited, which the report acknowledges.
Two: selection must be linear too. Whedon's critique of DeepSeek V3.2's Lightning Indexer is the sharpest argument in the report. A sparse attention that uses a small dense scorer to pick tokens is still quadratic, just with a smaller constant. Subquadratic says the indexer overtakes the attention it feeds at about 52K tokens, costing roughly 16x the attention compute at 1M and 190x at 12M. In a GLM 5.2 test, Whedon says 58% of prefill latency at 1M came from the selection step.
Three: replace every attention layer, not some. Hybrids such as Nvidia's Mamba based Nemotron or Qwen's Gated DeltaNet layers keep some dense attention. That is Amdahl's law in action: if 80% of layers go linear, the remaining 20% caps the speedup at 5x no matter how long the context gets. Full replacement is the only path to a 64.5x or 1,000x class number.
The Code, Annotated
SSA's selection mechanism is undisclosed, so no one outside the company can show you its code. What we can show is the general pattern every content dependent sparse attention follows, and run it.
import numpy as np
def dense_attention(q, K, V):
# Baseline: score the query against EVERY key. O(n) per query, O(n^2) per sequence.
s = K @ q / np.sqrt(q.shape[-1])
w = np.exp(s - s.max()); w /= w.sum()
return w @ V
def block_topk_attention(q, K, V, block=64, top_blocks=4, local=128):
"""Toy content dependent sparse attention. NOT SubQ's SSA, whose selection is
undisclosed. This is the public pattern: cheap coarse scoring, then exact
attention over a small subset."""
n, d = K.shape
nb = n // block
# 1) Coarse score with one mean pooled summary key per block: O(n/block), not O(n).
# A dense "indexer" that scores every key here is what makes DeepSeek style
# selection quadratic at scale. Keeping this step cheap is the whole game.
summaries = K[: nb * block].reshape(nb, block, d).mean(axis=1)
coarse = summaries @ q
# 2) Content dependent selection: keep the best blocks ANYWHERE in the sequence.
chosen = np.argsort(coarse)[-top_blocks:] # ← THIS is the trick: relevance, not distance
idx = np.concatenate([np.arange(b * block, (b + 1) * block) for b in chosen])
# 3) Always keep a local window. Recent tokens carry syntax and short range state.
idx = np.unique(np.concatenate([idx, np.arange(n - local, n)]))
# 4) Exact softmax over k = |idx| keys. Cost O(k), and k stays fixed as n grows.
s = K[idx] @ q / np.sqrt(d)
w = np.exp(s - s.max()); w /= w.sum()
return w @ V[idx], len(idx)
rng = np.random.default_rng(0)
n, d = 65_536, 64
K = rng.standard_normal((n, d)).astype(np.float32)
V = rng.standard_normal((n, d)).astype(np.float32)
needle = 41_000
q = K[needle] * 3 # the query strongly matches one buried key
K[needle - 32 : needle + 32] += K[needle] * 0.5 # its neighbors are related
dense = dense_attention(q, K, V)
sparse, k = block_topk_attention(q, K, V)
print(f"keys scored: dense={n:,} sparse={k:,} ({n/k:.0f}x fewer)")
print(f"cosine(dense, sparse) = {dense @ sparse / (np.linalg.norm(dense) * np.linalg.norm(sparse)):.4f}")
# Output (we ran it):
# keys scored: dense=65,536 sparse=384 (171x fewer)
# cosine(dense, sparse) = 1.0000
Caption: On a needle query, sparse attention matches dense exactly while reading 171x fewer keys. That is why needle in a haystack is the friendliest possible benchmark for this architecture.
Now the same toy on an aggregation query, where the answer is spread across 2,000 scattered tokens. This is the shape of RULER's common word and frequent word tasks, which is exactly where SubQ's RULER errors concentrate.
# Aggregation probe: no single needle, the answer lives in 2,000 scattered tokens.
rng2 = np.random.default_rng(1)
K2 = rng2.standard_normal((n, d)).astype(np.float32)
V2 = rng2.standard_normal((n, d)).astype(np.float32)
topic = rng2.standard_normal(d).astype(np.float32)
hits = rng2.choice(n, 2_000, replace=False)
K2[hits] += topic * 0.6
V2[hits] += 2.0
s = K2 @ topic / np.sqrt(d); w = np.exp(s - s.max()); w /= w.sum()
print(f"dense attention mass on answer tokens: {w[hits].sum():.3f}")
# → 0.969: dense attention looks at nearly all 2,000 answer tokens
nb = n // 64 # same selection as block_topk_attention above
summ = K2[: nb * 64].reshape(nb, 64, d).mean(1)
chosen = np.argsort(summ @ topic)[-4:]
idx = np.unique(np.concatenate([np.concatenate([np.arange(b*64, (b+1)*64) for b in chosen]),
np.arange(n - 128, n)]))
print(f"answer tokens seen by sparse: {len(set(idx) & set(hits))} of {len(hits)}")
# → 29 of 2000 ← THIS is the failure mode
# The output vector still has cosine 0.98 with dense, because averaged values look
# alike. A "count how many times X appears" question would still be wrong.
Caption: Sparse attention sees 29 of 2,000 answer tokens, 1.5%, while dense puts 96.9% of its weight on them. Output similarity hides the miss. Aggregation, not retrieval, is the stress test that matters.
It In Action
Input: a 1M token repository (roughly a mid sized monorepo) plus the question "Which services still call the deprecated auth_v1.verify()?" Prefill must attend over all 1M tokens before the first answer token.
Step one, per layer attention cost. Dense attention: 252 PFLOP. SSA: 3.9 PFLOP. Ratio 64.6x, matching the report's 64.5x.
Step two, per layer attention latency on one H100. FlashAttention-2: 54,164 ms. SSA: 966 ms. Ratio 56x.
Step three, scale to a full model. The layer count is undisclosed. Assuming a 61 layer donor (DeepSeek V3 and Kimi K2 both use 61, and outside researchers have guessed one of those families), attention alone is about 55 minutes dense versus 59 seconds sparse on a single GPU, before any parallelism. That is an assumption based estimate, not a reported number.
Step four, the cost that does not shrink. SSA still stores every token's keys and values, because selection must be able to reach any of them. Assuming DeepSeek V3 style Multi head Latent Attention (MLA, a compressed KV format of 576 values per token per layer), the KV cache is about 70 KB per token:
Context | KV cache (BF16, MLA assumption) |
|---|---|
128K | 9 GB |
1M | 70 GB |
2M | 141 GB |
12M | 843 GB |
Output: the attention bill drops from roughly an hour to roughly a minute. The memory bill at 12M exceeds an 8x H100 node's 640 GB of HBM.
Caption: SSA turns long context from a compute problem into a memory problem. That trade is worth it, but it is not free.
Why This Design Works, and What It Trades Away
Why it works. Attention is already sparse in practice. Whedon cites that over 99% of attention scores fall below 0.1, so computing them is mostly waste. SSA skips that waste without dropping tokens from the context, which is the difference from RAG (retrieval augmented generation, where a separate search step decides what the model ever sees). With SSA, the model itself decides, per token, per layer, using learned weights.
The cheapness is also self reinforcing. Whedon says the biggest driver of retrieval quality was routinely pretraining on very long sequences, which SSA made cheap enough to do: under one minute per step at 1M context, across 100+ experiments and six model generations.
What it trades away.
Short context speed. The report puts FlashAttention-2 parity near 16K tokens. Below that, SSA's selection overhead loses to dense. Most chat and agent turns live there.
Aggregation. Fixed k means a fixed number of things a query can look at. Counting, global statistics, and "summarize everything" tasks pay first.
Short context knowledge. The report says early long context gains regressed short context knowledge and needed explicit capability balancing.
Verifiability. The selection mechanism, donor model, and parameter count are all undisclosed. Outsiders cannot reproduce or audit the core idea.
Technical Moats
The moat is not the concept. Content dependent sparse attention has a long public lineage, from Longformer and BigBird through NSA and DeepSeek Sparse Attention. The moat is the training recipe for a transplant: knowing how to swap attention in a frontier donor, extend context in staged YaRN steps, alternate post training with recovery phases so short context skill survives, and run Ring Attention at million token steps without blowing the budget. Six model generations of that iteration is accumulated, private know how.
The second moat is the linear selector itself. If SSA really scores candidates in linear time without a dense indexer, it fixes the exact cost that dominates DeepSeek style designs past 52K tokens. That is the piece worth a technical paper, and it is the piece that has not been published.
Contrarian Insights
Insight One: the evidence proves retrieval at 12M, not reasoning at 12M. Every multi million token number is single needle UUID retrieval, "a relatively clean probe" in the report's own words. RULER, the composite test, is published only at 128K. MRCR, the multi needle test, was dropped as a primary signal; the May preview had reported 83 in research versus 65.9 in production. Our toy shows why this matters: on a needle, sparse matches dense perfectly; on aggregation, it sees 1.5% of the evidence. "12M context" today means "can find one thing in 12M tokens." Treat it as a search primitive, not a reading comprehension guarantee.
Insight Two: "linear compute and memory" undersells the real wall. Dense attention's KV cache was already linear in context length. SSA fixes compute, not storage. Under a plausible MLA donor, a 12M token context needs about 843 GB of KV cache per sequence, more than an 8x H100 node holds. The report also flags BF16 underflow beyond about 8M tokens. At the far end of the context curve, the binding constraints are memory capacity and numerics, and no attention trick removes either.
Surprising Takeaway
Divide each reported context length by its reported FLOP reduction and you get the same number every time: 32K / 2.1 = 15.6K, 128K / 8 = 16.4K, 512K / 31.5 = 16.6K, 1M / 64.5 = 16.3K, and 0.13% of 12M = 15.6K. SubQ appears to give each query a fixed budget of roughly 16K attended tokens. That single constant explains everything: the speedup grows linearly with length, parity with FlashAttention-2 lands near 16K where k equals n, and the "1,000x at 12M" figure works out to about 770x. SSA is a 16K attention model that can point anywhere in a 12M token window.
TL;DR For Engineers
SubQ 1.1 Small is a sparse attention transplant onto an unnamed open weight donor, plus ~1T tokens of long context training, not a new pretrain.
Reported numbers imply a fixed ~16K tokens attended per query, so cost is O(n·16K) and the win only starts past ~16K context.
Retrieval is proven to 12M (98% NIAH); composite reasoning is published only at 128K (RULER 99.12%).
Compute drops 64.5x at 1M, but KV memory still grows linearly; budget roughly 70 GB per 1M tokens under an MLA style donor.
Use it for whole repo and whole corpus lookup; keep testing aggregation and counting tasks before trusting it to summarize.
Explain It Like I'm New
Picture a lawyer handed ten thousand pages and one question. A careless lawyer reads every page and compares every sentence to every other sentence. That is how standard AI models read: each word checks itself against every other word. Double the pages and the work goes up four times.
That is why most AI tools quietly cut long documents into pieces and only show the model a few. The model never actually sees the whole thing.
SubQ takes a different approach. It keeps all ten thousand pages on the desk, but for each sentence it works on, it flips straight to the few hundred passages that matter and ignores the rest. The skill is in knowing where to flip, and the model learned that during training rather than relying on a separate search tool.
The result is that reading twice as much costs about twice as much work, not four times. That makes it practical to hand a model an entire codebase or a full legal archive in one go.
The catch is that flipping to the right pages is great for finding a specific fact and weaker for questions that need every page, like counting how often something happens. As AI systems move from answering questions to working over whole companies' worth of data, that difference between finding and reading will matter a lot.
See It In Action
SubQ-1.1-Small: Subquadratic Sparse Attention Explained (Theory Illustrated), YouTube. youtube.com/watch?v=g2Pk30gCgKI. A visual walkthrough of why dense attention is quadratic and how content dependent sparsity changes the curve.
What Is SubQ 1.1 Small and Subquadratic Sparse Attention, YouTube Shorts. youtube.com/shorts/sXWVrFCF9pg. A one minute primer, useful to share with non specialist teammates before the deep dive.
Subquadratic Preview Model Benchmark Evaluation, Appen. appen.com. The third party methodology: 50 samples per context tier at temperature 0 for NIAH, and 1,055 LiveCodeBench problems with four completions each.
SubQ 1.1 Small Technical Report and Model Card, Subquadratic. subq.ai. The primary source for every efficiency number in this issue, including the per layer FLOP and latency tables.
Community Conversation
Alex Whedon, CTO, via The New Stack, interview. Argues hybrids cap out near 5x, that DeepSeek style selection spends 58% of prefill at 1M on choosing tokens, and that the company is already working on architectures with no attention at all.
Will Depue, OpenAI researcher, quoted by The New Stack. Called SubQ "almost surely a sparse attention finetune of Kimi or DeepSeek" at launch. The model card later confirmed a donor model, which moves the question from "is it new" to "is the selector new."
Dan McAteer, AI commentator, via AI Wiki. Wrote that SubQ "is either the biggest breakthrough since the Transformer, or it's AI Theranos." It captures how much rides on independent reproduction.
DataCamp, SubQ explained. Flags that May launch benchmarks ran at 1M and 128K, not the claimed 12M, and compares the pitch to Magic.dev's 2024 100M token claim.
RuntimeWire, model card coverage. Frames the June release as the move from "provocative May launch claim" to "a more testable phase."
Fixed Budget Attention Is the Real Product
SubQ 1.1 Small does not read 12 million tokens the way a person reads a book. It indexes them and attends to about 16 thousand at a time, chosen by learned weights rather than a bolted on search engine. That is a real advance over RAG and over quadratic indexers, and the transplant recipe that got a Sonnet class donor there on a trillion tokens is the most underrated part of the report. But the industry should stop treating context length as one number. A model that can find any fact in 12M tokens and a model that can reason over 12M tokens are different products, and so far only the first one has been demonstrated.
References
SubQ 1.1 Small Technical Report and Model Card, Ramirez, Whedon, Vayani, Vo, Subquadratic AI. Primary source for SSA results.
RULER: What's the Real Context Size of Your Long Context Language Models?, Hsieh et al., NVIDIA. The 13 task suite behind the 99.12% score.
YaRN: Efficient Context Window Extension of Large Language Models, Peng et al. The context extension method used from 262K to 2M.
Ring Attention with Blockwise Transformers for Near Infinite Context, Liu, Zaharia, Abbeel. The final rung of SubQ's training memory ladder.
Native Sparse Attention, Yuan et al., DeepSeek. Hardware aligned, trainable sparse attention, the closest public precedent.
Longformer: The Long Document Transformer, Beltagy, Peters, Cohan. Local plus global sparse attention, the starting point of this lineage.
Lost in the Middle: How Language Models Use Long Contexts, Liu et al. Why a long window does not guarantee the model uses it.
SubQ 1.1 Small transplants a learned, linear time sparse attention into an open weight donor model and trains it on about a trillion long context tokens, cutting attention compute 64.5x at 1M tokens and retrieving needles at 98% out to 12M. Its reported numbers imply a fixed budget of about 16K attended tokens per query, which makes it an excellent long context search engine whose reasoning and memory costs at multi million lengths remain unproven.
Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀
Stop Paying for 6 Tools. One AI Does It All
Most e-commerce sellers are running their store across 6 to 8 separate tools — and paying hundreds of dollars a month for the privilege. StoreClaw replaces your entire stack with one autonomous AI engine that monitors competitors, optimizes listings, automates marketing, and tracks real profit across Shopify, Amazon, and beyond.
It doesn't wait for you to ask. It runs 24/7 in the background, so you wake up to a full dashboard instead of a list of things you forgot to check.
Connect your store, and StoreClaw gets to work — no prompts, no complex setup, no six-app stack.
Free to start. No credit card required.