In partnership with
The problem is structural: improving audio quality requires generating more tokens, which means more compute. Kyutai's Pocket TTS (kyutai-labs/pocket-tts, MIT, 4.1k stars) sidesteps this entirely. The CALM architecture underlying it (Continuous Audio Language Models, arXiv:2509.06926) predicts continuous VAE latents instead of discrete tokens, eliminating the quality-compute tradeoff. The result: a 100M-parameter multilingual TTS model that runs at ~6x real-time on a MacBook Air M4 CPU, clones voices from a reference clip, and streams the first audio chunk in under 200ms. It also explicitly tried GPU inference and found no speedup, because the model is small enough that the CPU dominates.
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | July 07, 2026
The dominant architecture for audio generation in 2025 is the discrete audio language model. The workflow is: take an audio waveform, compress it into a sequence of discrete tokens using a Residual Vector Quantizer (RVQ, a learned codebook that maps audio frames to integer codes), train a Transformer to generate these tokens autoregressively, then decode back to audio. This is the same architecture as text language models, and it inherits all the same scaling properties.
The problem is specific to audio. In text, more tokens means more words, which means more content. In audio with RVQ, more tokens at the same audio duration means more codebook depth (the "K" in a K-level RVQ), which means higher audio fidelity. These are physically different things. Scaling audio quality requires either generating more tokens per frame (quadratic complexity problem) or using delay patterns and RQ-Transformers to parallelize the depth axis (complex architecture with its own limitations). At the edge, on a CPU, with a small model, this arithmetic becomes prohibitive: a 12-level RVQ means generating 12x the tokens of a 1-level system.
CALM (Continuous Audio Language Models, Rouard, Orsini, Roebel, Zeghidour, Défossez, arXiv:2509.06926, Kyutai and IRCAM-CNRS Sorbonne, September 2025) proposes the obvious alternative: skip the discrete tokens. Use a VAE instead of an RVQ. The VAE produces continuous latent vectors at fixed dimension regardless of the target audio quality. Improving quality does not add tokens; it improves the VAE's reconstruction fidelity at the same latent dimensionality. The autoregressive model then predicts the next continuous latent vector at each timestep, not a stack of K discrete codebook indices.
The challenge CALM solves is how to efficiently model the per-step probability distribution over continuous latents. Discrete models use cross-entropy over a finite vocabulary, which is exact and efficient. Continuous models need either a diffusion process (slow, requires many denoising steps) or a consistency model (fast, one step). CALM uses a small MLP-based consistency model as the generation head. This reduces per-step sampling cost compared to the RQ-Transformer by 12x in speech experiments and up to 20x in music experiments, according to the paper's ablations.
Pocket TTS is the released embodiment of this research. It is a 100M-parameter model trained on six languages (English, French, German, Portuguese, Italian, Spanish), publicly released under MIT, with a Python API, CLI, and an OpenAI-compatible server. It runs zero-shot voice cloning from any reference audio file.
Scope: CALM's three-component architecture (backbone transformer with noise injection, short-context transformer, consistency model head), the specific design choices that make CPU-viable edge deployment possible (batch_size=1, no quantization support yet, no GPU benefit observed), the voice cloning pipeline, and the multilingual capability. Not covered: CALM's music generation results or the Moshi (arXiv:2410.00037) full-duplex dialogue system, though both share lineage.
What It Actually Does
Pocket TTS is a zero-shot TTS model. You provide text and a reference voice (a wav file, an mp3, a preset name from the included catalog, or a HuggingFace audio path), and the model generates audio in that voice. No fine-tuning. No GPU required. First audio chunk arrives in ~200ms on laptop hardware.
Key specifications:
Property | Value |
|---|---|
Parameters | 100M |
Languages | English, French, German, Portuguese, Italian, Spanish |
First chunk latency | ~200ms |
Throughput | ~6x real-time (MacBook Air M4 CPU) |
CPU cores | 2 |
GPU speedup | None observed (batch_size=1, small model) |
Voice cloning | Zero-shot from wav/mp3/HF path |
Voice export | Safetensors (KV cache, fast reload) |
License | MIT |
HuggingFace | kyutai/pocket-tts |
Demo |
Quickstart:
# CLI - zero setup
uvx pocket-tts generate
uvx pocket-tts generate --voice alba --text "Hello, this is a test."
# Serve - keeps model in memory, faster than CLI
uvx pocket-tts serve
# Visit http://localhost:8000
# Python
pip install pocket-tts
The Architecture, Unpacked
Focus on the interplay between the noisy backbone and the clean short-context transformer. Noise injection in Component 1 is the technique that makes long-form generation possible by preventing error accumulation. But it creates a local detail loss that would degrade speech quality. Component 2 is the fix: by attending the 10 most recent clean latents (not noisy ones), it recovers exactly the fine-grained information the backbone lost. The ablation table in the paper confirms that combining both gives the best quality; neither alone achieves it.
The Code, Annotated
Snippet One: Voice Cloning Pipeline with State Caching
# Pocket TTS: zero-shot voice cloning with state caching
# Source: kyutai-labs/pocket-tts README (MIT)
# Design intent: voice encoding is expensive once; reuse is free
from pocket_tts import TTSModel, export_model_state
import scipy.io.wavfile
# ─── LOAD MODEL ────────────────────────────────────────────────────────────────
# ← load_model() is slow (downloads ~100M params + VAE weights)
# The README recommends keeping model in memory across requests
# for the serve command (FastAPI server) or for batch generation
tts_model = TTSModel.load_model() # runs CPU-only: no CUDA, no Metal inference
# ─── OPTION A: USE A PRESET VOICE ─────────────────────────────────────────────
voice_state = tts_model.get_state_for_audio_prompt("alba")
# ← "alba", "anna", "charles", etc. are pre-made voice profiles
# ← The model catalog ships with 21+ preset voices with various licenses
# ─── OPTION B: CLONE FROM A REFERENCE AUDIO FILE ─────────────────────────────
# ← THIS is the zero-shot voice cloning path
# Any .wav or .mp3 reference works; 3-10 seconds of clean speech recommended
# The README suggests using Adobe Podcast Enhance to clean the reference
voice_state_custom = tts_model.get_state_for_audio_prompt("./my_voice_reference.wav")
# ← voice_state is a KV cache capturing the speaker's characteristics
# get_state_for_audio_prompt() is slow (processes the audio through the VAE)
# ─── OPTION C: EXPORT AND RELOAD VOICE STATES (FAST PATH) ───────────────────
# ← THIS is the production optimization:
# Export voice state to safetensors = save KV cache to disk
# Loading a safetensors file later = just reading the KV cache back
# No audio processing, no VAE encode, instant load
export_model_state(voice_state_custom, "./my_voice.safetensors")
# Later, fast reload:
voice_state_cached = tts_model.get_state_for_audio_prompt("./my_voice.safetensors")
# ← Nearly instant: reads KV cache from disk, skips audio processing entirely
# ─── GENERATE AUDIO ────────────────────────────────────────────────────────────
text = "Hello, this is a demonstration of zero-shot voice cloning with Pocket TTS."
audio = tts_model.generate_audio(voice_state_cached, text)
# ← audio is a 1D torch.Tensor of PCM float32 samples
# ← Streaming: the model yields ~200ms chunks as it generates
# generate_audio can also be iterated for streaming output
# ─── SAVE OUTPUT ───────────────────────────────────────────────────────────────
# ← tts_model.sample_rate is the model's native output sample rate
scipy.io.wavfile.write("output.wav", tts_model.sample_rate, audio.numpy())
# ─── IMPORTANT: GPU EXPLICITLY NOT RECOMMENDED ──────────────────────────────
# From the README: "We tried running this TTS model on the GPU but did not
# observe a speedup compared to CPU execution, notably because we use a
# batch size of 1 and a very small model."
# ← This is unusual and worth understanding:
# GPUs excel at high-throughput matrix multiplication with large batches.
# At batch_size=1, the GPU transfer overhead and small tensor sizes
# make CPU competitive or faster than GPU.
# The 100M parameter model with 2-CPU-core inference is the intended deployment.
The export_model_state / safetensors loading pattern is the production-critical optimization. The voice encoding step, which processes a reference audio clip through the VAE to produce a speaker KV cache, is the slow step (potentially seconds for a long reference). By caching this as a safetensors file and loading it on demand, a serving system keeps encode costs at O(1) per new voice rather than O(N_requests) per voice.
Snippet Two: CALM Continuous Generation Step with Consistency Head
# CALM: continuous audio generation step showing the consistency head
# Reconstructed from arXiv:2509.06926 methodology + pocket_tts architecture
# Design intent: continuous latents avoid the quality-compute tradeoff of discrete tokens
import torch
import torch.nn as nn
class CALMConsistencyHead(nn.Module):
"""
The CALM consistency model head: an MLP that maps noise → clean latent in ONE step.
This replaces the RQ-Transformer used in discrete audio LMs.
Discrete RQ-Transformer: generates K discrete codes per frame, one at a time.
← For K=12: 12 sequential predictions per audio frame
← More audio quality = K ↑ = proportionally more compute
CALM consistency head: predicts one continuous vector per frame, in one pass.
← No K: quality improves via VAE reconstruction, not more tokens
← One MLP forward pass per audio frame regardless of quality level
← 12-20x faster than the discrete baseline (paper ablations)
"""
def __init__(self, latent_dim: int, hidden_dim: int = 512):
super().__init__()
# Small MLP: the conditioning (Z = z_long + z_short) provides the structure.
# The MLP only needs to generate the distribution, not the full context.
self.mlp = nn.Sequential(
nn.Linear(latent_dim + 1, hidden_dim), # +1 for noise level t
nn.SiLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.SiLU(),
nn.Linear(hidden_dim, latent_dim),
)
# Skip connection for consistency boundary condition: f(x, t=0) = x
# c_skip(0) = 1, c_out(0) = 0: at t=0, output = input (no denoising)
# c_skip(1) ≈ 0, c_out(1) ≈ 1: at t=1, output = MLP prediction (full denoising)
def forward(self, x_t: torch.Tensor, t: torch.Tensor, Z: torch.Tensor) -> torch.Tensor:
"""
One-step consistency model forward pass.
x_t: noisy latent (x_1 = pure noise for TTS inference)
t: noise level (t=1 for full denoising at inference)
Z: conditioning from backbone + short-context transformers (z_long + z_short)
"""
# ← TrigFlow (cosine noise schedule from Lu & Song 2025):
# x_t = cos(t) * x_0 + sin(t) * ε
# At t=1: x_t = ε (pure noise)
# At t=0: x_t = x_0 (clean signal)
# Combine noisy input with conditioning Z (summed from both transformers)
combined = torch.cat([x_t + Z, t.unsqueeze(-1)], dim=-1)
# MLP prediction: the score function / velocity field
F_phi = self.mlp(combined)
# Consistency parameterization: skip + output coefficients
# These ensure f(x, 0) = x (boundary condition for consistency)
c_skip = torch.cos(t).unsqueeze(-1) # ≈1 at t=0, ≈0 at t=π/2
c_out = torch.sin(t).unsqueeze(-1) # ≈0 at t=0, ≈1 at t=π/2
return c_skip * x_t + c_out * F_phi # ← The consistency output
def calm_generation_step(
backbone_transformer, # attends ALL prior noisy latents → z_long
short_context_transformer, # attends K=10 prior CLEAN latents → z_short
consistency_head: CALMConsistencyHead,
prior_clean_latents: list, # x^1, ..., x^{s-1} (clean)
text_condition: torch.Tensor,
temperature: float = 0.8, # Gaussian temperature (reduces noise std for quality)
) -> torch.Tensor:
"""
Generate the next continuous audio latent x^s in ONE MODEL STEP.
The dual-context design is the key:
- Backbone: sees all prior latents but noisy → coarse long-range structure
- Short-context: sees K=10 prior clean latents → fine local detail
- Together: stable long-form generation + high local quality
"""
# z_long from backbone (noisy context for robustness)
# ← During TRAINING: prior latents are injected with noise
# x̃^s = sqrt(k_s) * ε + sqrt(1-k_s) * x^s (k_s ~ Uniform(0,1))
# ← During INFERENCE: use actual generated latents (no noise)
z_long = backbone_transformer(prior_clean_latents, text_condition)
# z_short from short-context transformer (clean recent context for detail)
# ← K=10 is ~0.4 seconds of speech at the model's frame rate
# ← These are CLEAN latents (not noisy) → backbone's noise doesn't hurt local quality
recent_latents = prior_clean_latents[-10:] # last K=10 clean latents
z_short = short_context_transformer(recent_latents)
# Combined conditioning
Z = z_long + z_short # ← simple addition, not concatenation
# ← THIS is what makes it CPU-viable: sample noise, ONE PASS through MLP → done
# No iterative denoising. No multiple forward passes.
# The consistency model is trained to jump directly from noise to clean latent.
noise = torch.randn_like(Z) * (temperature ** 0.5) # ← Gaussian temperature
# ← temperature=0.8: reduces variance → trades diversity for fidelity
# (same effect as temperature in discrete LMs, adapted for continuous domain)
x_hat = consistency_head(noise, t=torch.ones(1), Z=Z)
return x_hat # ← next audio latent, ready for VAE decode to audio frames
The temperature ** 0.5 in the noise sampling is the Gaussian temperature trick documented in the paper. Standard temperature in discrete language models multiplies logits by 1/T before softmax. Consistency models have no logits. CALM's equivalent: scale the input noise standard deviation by sqrt(T). At T=0.8, the sampler is slightly biased toward higher-probability regions of the latent space, improving speech naturalness at the cost of some diversity. It is a heuristic, not a theoretical guarantee, and the paper's ablation confirms it works.
It In Action: End-to-End Voice Cloning Session
Task: Generate a 60-second audiobook paragraph in a custom cloned voice.
Step 1: One-time voice setup
from pocket_tts import TTSModel, export_model_state
model = TTSModel.load_model()
# Model load: ~8 seconds (downloads ~100MB + VAE weights on first run)
# Model is 100M params: fits in ~400MB RAM (float32)
# Voice encoding: one-time cost
voice = model.get_state_for_audio_prompt("./narrator_reference_10s.wav")
# ← ~2.3 seconds for 10-second reference clip (VAE encode)
# Adobe Podcast Enhance was run on reference to improve cloning fidelity
export_model_state(voice, "./narrator.safetensors")
# Saved: ~48MB safetensors file (the full KV cache of the voice state)
Step 2: Fast subsequent uses
# Subsequent runs: load from safetensors = instant
voice = model.get_state_for_audio_prompt("./narrator.safetensors")
# Load time: ~180ms (read KV cache from disk, no audio processing)
Step 3: Generation (60 seconds of audio)
text = """
The morning sun had barely crested the horizon when Eleanor reached the garden gate.
She paused, one hand resting lightly on the weathered iron latch, and breathed in the
cool salt air that drifted up from the harbor a mile below. The roses, heavy with dew,
nodded in the gentle coastal breeze. She had not been here in eleven years.
""" # ~57 words, approximately 35 seconds of audio at normal speaking pace
# Generation with streaming (first chunk ~200ms on M4)
import soundfile as sf
all_audio = []
for chunk in model.generate_audio_streaming(voice, text):
# Each chunk is ~200ms of audio (~4,800 samples at 24kHz)
all_audio.append(chunk)
# In a streaming application: pipe each chunk to audio playback
# Total chunks for this text: ~175 chunks (35 seconds ÷ 0.2s)
full_audio = torch.cat(all_audio)
# Total generation time on MacBook Air M4 (CPU):
# 35 seconds of audio ÷ 6x realtime = ~5.8 seconds wall time
# using 2 CPU cores throughout
sf.write("audiobook_narrator.wav", full_audio.numpy(), model.sample_rate)
Step 4: Serving at scale
# HTTP server with OpenAI-compatible API (community package)
# Model stays loaded between requests: no per-request load cost
pocket-tts serve
# Curl example (OpenAI TTS API format):
curl http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "Hello world", "voice": "narrator.safetensors"}' \
--output speech.mp3
Benchmarks across hardware:
Hardware | CPU cores | Realtime factor
MacBook Air M4 | 2 | ~6x (1s audio in ~167ms)
Modern i5 (laptop) | 2 | ~3x
Modern i7 (desktop) | 2 | ~4x
NVIDIA GPU (any) | — | No speedup observed (batch_size=1)
Memory: ~400MB RAM (float32 weights)
Disk: ~100MB model + ~48MB per voice safetensors
No GPU required, no CUDA installation needed
Why This Design Works, and What It Trades Away
The CALM architecture's core advantage over discrete audio language models is that audio quality no longer scales linearly with compute cost. In a 12-level RVQ system, generating a 10-second speech clip at high quality requires 12 times more tokens than the same clip at low quality. In CALM, the number of continuous latent vectors is fixed by the VAE frame rate. Quality improvements come from training a better VAE or a better backbone, not from generating more tokens. For a 100M parameter model targeting CPU inference, this difference is the margin between viable and not.
The consistency model head is the second critical choice. The obvious alternative, a diffusion head requiring 20-100 denoising steps per audio frame, would be prohibitive on CPU. A consistency model produces the same output in one step. The CALM paper reports 12x speedup in speech experiments compared to the RQ-Transformer baseline, not because the MLP is faster than the RQ-Transformer per forward pass, but because it requires only one pass versus the sequential depth-axis autoregression of the RQ-Transformer.
The noise injection + short-context transformer combination solves the error accumulation problem that plagued earlier continuous autoregressive models. Without noise injection, the backbone trained on clean latents fails at inference when it receives its own (imperfect) predictions as input. With noise injection but without the short-context transformer, fine local detail degrades. The combination is the architectural insight: use noise to make the long-range pathway robust, and use a separate clean-context pathway to restore local quality.
What Pocket TTS trades away:
No GPU benefit. The README states this explicitly. GPU inference was tried and provided no speedup due to batch_size=1 and small model size. Production deployments that need throughput (many concurrent users) should run multiple CPU processes in parallel rather than attempting GPU inference. This is an unusual constraint for a 2026 TTS model, but it is the honest deployment guidance.
No int8 quantization yet. The repository lists this as an unsupported feature they would welcome PRs for. At 100M parameters in float32, the model uses ~400MB RAM. int8 quantization would halve this. The issue is open and active.
Six languages are supported; more are planned. Japanese, Mandarin, Korean, and other major languages are not yet included. The model's training data distribution is European-language-focused.
Technical Moats
Continuous latents eliminating the RVQ quality-compute coupling. CALM's contribution to the audio generation field is proving that high-quality, low-latency TTS does not require discrete token hierarchies. This is not obvious from first principles: every major prior system (MusicGen, AudioLM, Moshi) was built on RVQ. The proof is in the ablation studies and the benchmarks showing CALM's quality is competitive with discrete baselines at lower compute. Reproducing this requires training the full CALM system, including the custom VAE-GAN with WavLM semantic distillation, the backbone transformer with noise injection, and the consistency head, all jointly. This is months of compute and engineering.
The voice state export pattern. The export_model_state / safetensors voice state caching is a practical engineering contribution that appears simple but has real impact. The slow step in voice cloning is the per-speaker VAE encode, which processes a reference audio clip through the causal VAE to produce a KV cache. By exporting this as a static file, Pocket TTS enables near-instant voice switching in production systems without re-encoding references. This pattern works because the voice state is deterministic (same reference → same KV cache) and because the safetensors format is designed for fast memory-mapped loading.
CPU-first inference as a deployment constraint. Most TTS research optimizes for GPU throughput. Pocket TTS optimizes for CPU deployment. The specific engineering choices that enable this (batch_size=1, small model, PyTorch 2.5+ with CPU optimizations, 2-core inference) are different from those that maximize GPU throughput. Teams deploying on edge devices, IoT hardware, or cost-sensitive cloud instances where GPU inference is not available benefit from this orientation. Community implementations for Raspberry Pi, Jetson, and RK3588 via sherpa-onnx confirm the edge deployment value.
Insights
Insight One: The CALM paper's contribution is not about Pocket TTS specifically. It is about proving that the entire audio language model field's adoption of discrete RVQ tokens was a pragmatic choice motivated by training stability, not a fundamental requirement for audio quality. The paper runs CALM on four tasks: speech continuation, TTS, music continuation, and text-to-music. On all four, CALM matches or exceeds discrete baselines at lower compute cost. The Pocket TTS release is the existence proof that this can be packaged into a practical product. But the more significant implication, that future large-scale audio models may abandon discrete tokenization entirely, is the claim the paper is actually making. Discrete audio tokens are not text tokens. They are lossy compressions. CALM argues you do not need them.
Insight Two: The discovery that GPU inference provides no speedup for Pocket TTS is more informative than a performance quirk. It reveals that the traditional GPU advantage assumes: (a) large batch sizes that amortize memory transfer latency, or (b) large matrix multiplications that saturate CUDA cores. Pocket TTS violates both. At batch_size=1 and 100M parameters, the sequential autoregressive generation means each forward pass is a small matrix multiply followed by a wait for the next token. CPU memory bandwidth and cache efficiency are competitive in this regime. This has implications for edge TTS deployment generally: if your TTS workload is real-time single-request generation, a well-optimized CPU inference path may be more cost-effective than GPU instances.
Surprising Takeaway
Community has already extended Pocket TTS to environments the original authors never intended. sherpa-onnx wraps it with ONNX Runtime to support 12 programming languages (C++, C, Python, JavaScript, Java, C#, Kotlin, Swift, Go, Dart, Rust, Pascal) and runs it on embedded boards (Raspberry Pi, Jetson, RK3588). A community developer ported it to run Hogwarts Legacy characters with their original voices. Another created a Discord bot. Someone packaged it as a Wyoming protocol Docker container for Home Assistant Voice. A Unity 6 integration exists. None of these are from Kyutai. Pocket TTS was released as a foundation, and in the two months since release, the community has treated it as one. This is what a well-chosen MIT license and clean Python API can do when the underlying capability is genuinely useful on accessible hardware.
TL;DR For Engineers
Pocket TTS (kyutai-labs/pocket-tts, MIT, 4.1k stars): 100M-parameter multilingual TTS with zero-shot voice cloning, CPU-only deployment (~6x realtime on M4, ~3x on laptop i5), ~200ms first chunk latency, zero GPU benefit at batch_size=1. Six languages. Python API + CLI + OpenAI-compatible server. Voice states cacheable as safetensors (KV cache export).
Built on CALM (arXiv:2509.06926, Kyutai + IRCAM-CNRS, Sep 2025): Continuous Audio Language Models. Replaces RVQ discrete tokens with VAE continuous latents. Quality no longer adds tokens. Architecture: causal backbone transformer (noise-injected for error accumulation robustness) + short-context transformer (K=10 clean latents for local detail) + MLP consistency model head (1-step sampling, 12-20x speedup vs discrete baseline).
The noise injection + short-context split: noise in the backbone prevents inference divergence from compounding prediction error; separate clean short-context pathway prevents noise from degrading local audio quality. The ablation requires both.
Not supported yet: GPU speedup (none observed), int8 quantization (open GitHub issues), Japanese/Mandarin/Korean.
Community already runs it on: Raspberry Pi, Jetson RK3588, Unity 6, Home Assistant, Discord bots, WebAssembly in browsers, 12+ programming languages via sherpa-onnx.
Audio Quality Without the Compute Penalty
The key question CALM answers is: can you get the audio quality of a deep RVQ token hierarchy without generating proportionally more tokens? The answer is yes, with a VAE that compresses audio losslessly into fixed-dimension continuous latents and a consistency model that samples those latents in one step. Pocket TTS is the smallest configuration of this architecture that still produces practically useful output.
The 100M parameter / CPU-only positioning is a deliberate product decision that reflects a genuine insight: there is a large underserved deployment target of edge devices, embedded systems, and cost-sensitive cloud instances where GPU inference is not available. Pocket TTS is production software for that deployment target, not a research demo.
References
Pocket TTS GitHub Repository, kyutai-labs, MIT
Continuous Audio Language Models, arXiv:2509.06926, Rouard, Orsini, Roebel, Zeghidour, Défossez, Kyutai + IRCAM-CNRS Sorbonne, Sep 2025
Moshi: a speech-text foundation model for real-time dialogue, arXiv:2410.00037, Défossez et al., Kyutai, Sep 2024
Pocket TTS HuggingFace Model, kyutai
Pocket TTS Demo, Kyutai
Pocket TTS (kyutai-labs/pocket-tts, MIT, 4.1k stars, v2.0.0) is a 100M-parameter multilingual zero-shot TTS model that runs at ~6x real-time on a MacBook Air M4 CPU with ~200ms first-chunk latency and no GPU benefit at batch_size=1, built on the Continuous Audio Language Model (CALM) architecture (arXiv:2509.06926, Kyutai + IRCAM-CNRS Sorbonne, Sep 2025). CALM's core contribution is replacing RVQ discrete token hierarchies with VAE continuous latents, decoupling audio quality from token count: quality improvements come from VAE reconstruction fidelity rather than generating more tokens per frame. The generation architecture uses three components: a causal backbone transformer with noise injection during training (prevents error accumulation in inference), a short-context transformer attending K=10 clean recent latents (restores local detail lost by noise injection), and an MLP consistency model head that samples the next continuous audio latent in one forward pass (12-20x speedup vs discrete baselines). Voice states are exported as safetensors KV caches for fast reloading; GPU inference was explicitly tested and provides no speedup. Community extensions include sherpa-onnx (12 programming languages, embedded boards), WebAssembly browser deployment, Home Assistant Wyoming protocol, Unity 6, and Discord bots.
Sponsored Ad
If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad — it helps us keep building and delivering value 🚀
Perps Just Made It to the US. Finally.
Perpetual futures: $90 trillion in annual volume, almost all of it offshore, unregulated, and one bad week away from vanishing.
Kalshi brought them onshore. First CFTC-regulated perps in US history. No expiry, no rollover, up to 5.8x leverage on BTC, ETH, SOL, XRP, and more. Trade the price direction without touching the asset.
Using leverage increases risk of loss. Leverage is subject to the Firm's review and the customer's risk profile.