In partnership with
Emotion sliders get the headlines. Memory bandwidth decides where Zonos can run, and why its Mamba hybrid barely changes that.
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 24, 2026
The promise: by the end of this issue you will understand how Zonos turns a short recording into expressive 44 kHz speech, what its eleven conditioning knobs really control, and the hardware math that says where it can and cannot run in real time.
The fence: we cover Zonos-v0.1 in both its transformer and hybrid variants, Zyphra's release post, and the two papers underneath: the DAC codec and Mamba-2. Zyphra's newer ZONOS2 gets a short section, not a teardown. We do not score voice quality ourselves.
What It Actually Does
Zonos-v0.1 is an open weight text to speech model from Zyphra, released in February 2025 under Apache 2.0. Give it text and a voice sample of a few seconds (Zyphra recommends 10 to 30), and it speaks the text in that voice at 44.1 kHz, with controls for emotion, pitch variation, speaking rate and audio quality.
Property | Value |
|---|---|
Variants | Transformer (26 attention layers) and hybrid (34 Mamba2 + 12 attention layers) |
Size | 1.62B autoregressive parameters by our count; ~1.6B per Zyphra for each variant |
Training data | 200,000+ hours, mostly English, plus Chinese, Japanese, French, Spanish, German |
Audio codec | DAC 44.1 kHz: 9 codebooks × 1,024 entries at 86 frames per second |
Max length | 30 seconds per generation |
Speed (Zyphra) | ~2x real time on an RTX 4090; hybrid ~20% faster |
Hardware | 6 GB+ VRAM; hybrid needs an RTX 3000 series or newer |
Text frontend | eSpeak phonemizer, 109 language codes accepted |
There is a Gradio UI, a Docker image, and a hosted playground. The repository has been quiet since March 2025; Zyphra's attention has moved to ZONOS2 (more below).
The Architecture, Unpacked
Zonos is a classic codec language model. Audio becomes a grid of discrete tokens; a 1.6B backbone predicts that grid one 11.6 ms step at a time; a neural codec turns tokens back into sound.
Caption: Every control lives in the prefix. The backbone sees one sequence: conditioning tokens, then audio tokens, and must run twice per step for classifier free guidance.
Four design decisions do the work.
One: discrete codes from a high fidelity codec. DAC compresses 44.1 kHz audio into 9 residual codebooks of 1,024 entries at 86 frames per second. That is 7.75 kbps, enough for broadcast quality, but it means 86 sequential model steps per second of speech.
Two: the delay pattern. Predicting all 9 codebooks of a frame at once ignores that fine codebooks depend on coarse ones. Shifting codebook k by k+1 steps lets each step predict the coarse code of the current frame alongside finer codes of earlier frames, all 9 in parallel, at the cost of 8 extra steps per generation.
Three: everything is a prefix condition. Speaker, emotion (an 8D vector), maximum frequency, pitch standard deviation, speaking rate, and language are each embedded and prepended. Each has a learned "unconditional" value, which is what makes classifier free guidance and partial control possible.
Four: a hybrid for memory. The hybrid swaps 34 of 46 layers for Mamba2 state space blocks, which keep a constant size state instead of a growing KV cache. Zyphra says it is the first TTS model built on a hybrid transformer and Mamba architecture.
The Code, Annotated
First, the real API, from the README and conditioning.py. The conditioning dict is the product.
from zonos.model import Zonos
from zonos.conditioning import make_cond_dict
import torchaudio
model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device="cuda")
wav, sr = torchaudio.load("me.wav")
speaker = model.make_speaker_embedding(wav, sr) # 128d, any length input
cond = make_cond_dict(
text="The quarterly numbers are in, and they are better than expected.",
speaker=speaker, language="en-us",
emotion=[0.6, 0.05, 0.05, 0.05, 0.1, 0.05, 0.05, 0.05], # happiness first
pitch_std=60.0, # 20 to 45 normal, 60 to 150 expressive
speaking_rate=15.0, # phonemes per second
fmax=22050.0, # recommended for cloning
)
codes = model.generate(model.prepare_conditioning(cond)) # ≤ 86 × 30 steps
audio = model.autoencoder.decode(codes) # 44.1 kHz
Caption: The repo's own notes say emotion is entangled with the text's sentiment and with pitch_std. Angry text sounds angry more easily than sad text sounds angry.
Second, the delay pattern, verbatim from zonos/codebook_pattern.py. Two lines carry the whole multi codebook strategy.
def apply_delay_pattern(codes, mask_token):
codes = F.pad(codes, (0, codes.shape[1]), value=mask_token) # add 9 steps of padding
return torch.stack([codes[:, k].roll(k + 1) for k in range(codes.shape[1])], dim=1)
# codebook 0 shifts 1 step, codebook 8 shifts 9 steps:
# at step t the model emits coarse code for frame t-1 and finest code for frame t-9
def revert_delay_pattern(codes):
_, n_q, seq_len = codes.shape
return torch.stack([codes[:, k, k + 1 : seq_len - n_q + k + 1] for k in range(n_q)], dim=1)
Caption: A frame is complete only 9 steps (about 104 ms) after its coarse code appears. That is the floor on streaming latency before any compute.
Third, our sizing and CPU measurement. We instantiated the exact v0.1 transformer backbone from the published config.json (random weights, bf16) and timed one decode step at batch 2, as classifier free guidance requires.
bb = TorchZonosBackbone(BackboneConfig(d_model=2048, n_layer=26,
attn_mlp_d_intermediate=8192, attn_layer_idx=list(range(26)), attn_cfg=gqa))
# transformer backbone 1.581B; per block 60.8M (attention 10.5M, MLP 50.3M)
# + 9 embeddings + 9 heads 37.8M -> 1.62B autoregressive parameters
# hybrid: 34 Mamba2 (~25.8M each) + 12 attention blocks -> ~1.61B
# bf16 weights read per step 3.24 GB -> 279 GB/s for real time
# RTX 4090 (1008 GB/s): ceiling 3.61x real time
# RTX 3060 (360 GB/s): ceiling 1.29x
# Apple M4 (120 GB/s): ceiling 0.43x
# DDR5 dual channel (~80 GB/s): ceiling 0.29x
# KV per step: 52 KiB transformer, 24 KiB hybrid (x2 for CFG)
# measured, 2 vCPU Xeon, bf16: 362 ms per step -> 0.032x real time
Caption: Zyphra's ~2x on a 4090 sits at 55% of the bandwidth ceiling, typical for batch 2 decoding. No amount of kernel work makes the unquantized model real time on an ordinary CPU.
It In Action
Input: a 24 second voice clip, a 50 word sentence, default settings, as in The Register's hands on test.
Step one, speaker embedding. The clip is resampled to 16 kHz, turned into log filterbanks, and pooled to a single 128 number vector. Length does not matter; Zyphra suggests concatenating several clean clips.
Step two, prefix. Phonemes plus six more conditioning embeddings form the prefix, prefilled once for both the conditional and unconditional branches.
Step three, decode. 50 words is about 20 s of speech, so 1,723 frames + 8 delay steps. Each step runs the 1.62B model on a batch of 2, about 6.5 GFLOP, and reads 3.24 GB of weights. By the end, the KV cache holds about 180 MB across both branches.
Step four, codec. DAC decodes the 9 × 1,723 code grid into 44.1 kHz audio.
Output: about 20 seconds of cloned speech.
Hardware | Time for ~20 s of audio | Source |
|---|---|---|
RTX 4090 | ~10 s (~2x real time) | Zyphra |
RTX 6000 Ada | 9 to 10 s for 18 to 20 s | The Register, hands on |
Apple M4, bf16 (bandwidth ceiling) | ≥ 46 s | SnackOnAI math |
2 vCPU cloud Xeon, bf16 | ~10.4 min (backbone alone) | SnackOnAI measurement |
Caption: The same model is a real time tool on a gaming GPU and an overnight batch job on a laptop CPU. The divide is memory bandwidth.
Why This Design Works, and What It Trades Away
Why it works. Codec language models inherit what made LLMs strong: they learn style, emotion, breathing and room tone directly from 200,000 hours of data, with no hand built duration or pitch model. Prefix conditioning plus learned unconditionals turns metadata into knobs, and classifier free guidance sharpens adherence to them. DAC's 44.1 kHz output avoids the muffled quality of 16 to 24 kHz systems.
What it trades away.
Compute per second of audio. 86 sequential steps, each a full 1.6B pass, doubled for guidance: 0.56 TFLOP and 279 GB of weight reads per second of speech.
Length. A 30 second cap per generation; long text must be split, and the docs warn that squeezing too many phonemes into 30 s pushes the rate knob out of distribution.
Control precision. Knobs are correlated training metadata, not independent dials. Emotion couples with pitch variation and with the VQScore and DNSMOS quality targets, which favor neutral speech.
Language claims. eSpeak accepts 109 codes, but training covered six languages. The others will phonemize and then sound wrong.
Hybrid portability. The Mamba2 path needs
mamba-ssmkernels and an Ampere or newer NVIDIA GPU; no CPU, no Apple Silicon.
Technical Moats
Data scale for an open model. 200,000+ hours of varied, expressive speech, sourced from the web according to Zyphra. Expressiveness at this level comes from data breadth more than from architecture.
A rich conditioning interface. Eleven conditioners in the hybrid (speaker, emotion, fmax, pitch, rate, language, VQScore per eighth of the clip, CTC loss, DNSMOS, speaker noising), each with a learned unconditional. Few open TTS models expose this much control.
Hybrid SSM experience. Zyphra builds Mamba hybrids for language models too, and Zonos was a public proof that the recipe transfers to audio.
Apache 2.0 at 1.6B. A permissive license on a model this expressive lets teams fine tune, host, and ship it commercially without negotiation.
Contrarian Insights
Insight One: the Mamba hybrid cannot be much faster, and our math shows why. State space layers shine when attention's KV cache dominates. In Zonos, it does not. At the 30 second limit, the transformer's whole KV cache across both guidance branches is about 276 MB, against 3.24 GB of weights read on every step. Even at maximum length, reading KV is under a tenth of per step memory traffic, and the hybrid still reads roughly the same 1.6B weights. That is why the measured gain is about 20%, smaller still on short prompts. The hybrid's real advantages are a constant memory footprint and cheaper very long contexts, which the 30 second cap mostly rules out.
Insight Two: emotion control is a property of the data, not a feature of the model. The conditioning docs say plainly that emotion is entangled with text sentiment, pitch variation, and quality scores. The most effective trick they describe is "negative prompting": run guidance against a strongly neutral emotion vector instead of the true unconditional. That is a sampling hack, not a disentangled control. Teams building products on "set emotion to 0.8 sad" should budget for per voice tuning and human review.
Surprising Takeaway
To talk in real time, Zonos must stream about 279 GB of weights per second, roughly 65 times what Pocket TTS needs. That puts a hard ceiling of about 0.3x to 0.4x real time on typical laptop memory, before any software inefficiency. For codec language models, the spec that decides "can it run here" is memory bandwidth, not FLOPs or VRAM.
TL;DR For Engineers
Zonos-v0.1: 1.62B autoregressive model over DAC 44.1 kHz tokens (9 codebooks, 86 Hz, delay pattern), Apache 2.0, transformer and Mamba2 hybrid variants.
Every step runs at batch 2 for CFG and reads ~3.24 GB in bf16: ~279 GB/s for real time; a 4090 tops out around 3.6x, laptops below 0.5x.
Our 2 vCPU CPU test: 362 ms per step, 0.032x real time. Use a GPU or a hosted endpoint.
Hybrid saves KV memory (24 vs 52 KiB per step) but not weight traffic; expect ~20% speedups, not multiples.
Conditioning knobs are entangled; start from pitch_std 20 to 45, speaking_rate ~15, fmax 22050, and tune per voice.
Explain It Like I'm New
Imagine a skilled impressionist who has listened to two hundred thousand hours of people talking: happy, angry, whispering, reading the news. Play them half a minute of your voice, hand them a script, and say "read this cheerfully, a bit faster." Zonos is a computer program that does that.
It works by treating sound like text. A compression tool, called a codec, turns audio into a long list of code numbers, about 86 tiny slices per second, with nine numbers per slice. Zonos learned to predict those numbers one slice at a time, the way a chatbot predicts the next word. Settings like emotion, pitch, and speed are given at the start, like stage directions before a scene.
The cost is in the repetition. For every slice of sound, the program has to read its entire memory of 1.6 billion learned numbers, and it does that twice, once with your directions and once without, comparing the two to follow your directions more closely. That adds up to moving hundreds of gigabytes of data every second.
A gaming graphics card can do that faster than you can listen. An ordinary laptop processor cannot, no matter how clever the software. That is why Zonos sounds remarkable but is not something most people can run on a regular computer.
See It In Action
Zyphra Audio Playground, Zyphra. playground.zyphra.com/audio. The hosted version, with cloning and emotion controls, no GPU needed.
Zonos release post and samples, Zyphra. zyphra.com/post/beta-release-of-zonos-v0-1. Architecture overview and audio comparisons.
Conditioning guide, Zyphra on GitHub. CONDITIONING_README.md. The most useful document in the repo: what each knob does and how they interact.
Gradio interface, Zyphra on GitHub. gradio_interface.py. Run locally to compare transformer and hybrid on your own voice.
Community Conversation
The Register, hands on with Zonos cloning. A 24 second clip fooled close friends and family "at first blush," though pacing felt slightly off; the hybrid sounded more polished, which "ironically took away somewhat the authenticity." The reviewer urges not exposing the Gradio UI publicly and flags impersonation risks.
MarkTechPost, release coverage. Summarizes Zyphra's claim that early evaluations match or beat ElevenLabs and Cartesia, without published scores.
The Decoder, Zonos can clone your voice and is open source. Frames the release as open weights catching up with proprietary voice cloning.
r/LocalLLaMA via Prismix, ZONOS2 discussion. The successor: an 8B sparse MoE with ~900M active parameters, trained on 6M+ hours, reading raw UTF-8 bytes instead of eSpeak phonemes, also Apache 2.0.
On consent: cloning from seconds of audio makes impersonation cheap. Clone only voices you have permission to use, and keep local demo servers off the public internet.
What ZONOS2 Says About This Design
Zyphra's own successor answers several of v0.1's weaknesses. Dropping eSpeak for byte level input fixes the phonemizer bottleneck for low resource languages and code switching. Scaling data 30x targets hallucinations and repetitions. And a sparse MoE with ~900M active parameters cuts per step weight traffic relative to v0.1's 1.6B dense pass, which is exactly the lever our bandwidth math points to. We have not verified ZONOS2's benchmark claims; it deserves its own issue.
Codec Language Models Are Bandwidth Machines
Zonos-v0.1 is a well engineered codec language model: high fidelity DAC tokens, a delay pattern for parallel codebooks, rich prefix conditioning, and a credible Mamba hybrid. Its quality comes from data and its cost comes from bandwidth. Each second of speech requires 86 full passes over 1.6B parameters, twice. That is why it shines on a 4090, struggles on a laptop, and why the hybrid's gains are modest. If you choose an expressive cloning model for production, size the hardware by gigabytes per second, and expect the next generation to win by activating fewer parameters per step.
References
High-Fidelity Audio Compression with Improved RVQGAN, Kumar et al. The DAC codec: 44.1 kHz audio at 8 kbps with residual vector quantization.
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, Dao and Gu. Mamba-2, the layer in the hybrid backbone.
Simple and Controllable Music Generation, Copet et al. MusicGen, which introduced the codebook delay pattern Zonos uses.
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers, Wang et al. VALL-E, the template for prompting a codec LM with a short voice sample.
Classifier-Free Diffusion Guidance, Ho and Salimans. The guidance scheme that doubles every Zonos decode step.
Zonos-v0.1 is Zyphra's Apache 2.0, 1.6B parameter codec language model that clones voices from seconds of audio and exposes emotion, pitch and rate controls, in transformer and Mamba2 hybrid variants. Our math shows real time playback needs about 279 GB/s of weight reads, which explains its GPU dependence and the hybrid's modest speedup.
Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀
Why Most AI Tools Fail at B2B Customer Service
B2B customer issues rarely resolve in a single interaction. They move across teams, systems, and account history that most AI tools were never designed to track.
A new briefing paper from Harvard Business Review Analytic Services, sponsored by Front, looks at why AI built for high-volume consumer support creates new coordination gaps when it meets the multi-team reality of B2B service, and what leaders should ask before investing in their next AI tool.
Get the free briefing paper to see the three questions that separate tools built for your environment from tools that will create more work.