Sponsored by
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | September 02, 2026
The Promise
The library's own benchmark file tells you which quantization format to pick, and for most serving workloads the answer is not the smallest one.
What this covers: where the six techniques actually sit, why 4-bit inverts with batch size, what breaks between weight and activation quantization, and how AutoQuantize's sensitivity search works. What this excludes: speculative decoding, diffusion and ONNX paths, each worth its own issue.
What It Actually Does
NVIDIA Model Optimizer is an Apache 2.0 library bundling six compression techniques behind one Python API: post-training quantization, quantization-aware training and distillation, pruning, knowledge distillation, speculative decoding and sparsity. Input is Hugging Face, PyTorch or ONNX. Output is a checkpoint that runs on TensorRT-LLM, vLLM or SGLang.
It went open source in January 2025 and dropped "TensorRT" from its name in December 2025, which matters more than a rebrand usually does. The library now exports to runtimes NVIDIA does not own.
The proof points are real and recent. Nemotron 3 Ultra at 550B quantized to NVFP4 claiming up to 5.9x higher decode-heavy throughput than GLM-5.1 754B FP4 while matching BF16 accuracy. Bielik.AI built a Minitron 7B that is 33% smaller and 50% faster retaining 90% quality. Domyn compressed Colosseum from 355B to 260B. Adobe cut diffusion latency 60% and total cost of ownership 40%.
Now open examples/benchmark.md, which NVIDIA ships in the repo, and read the PTQ table.
Llama3.1-8B, H200 | batch 1 | batch 8 | batch 64 |
|---|---|---|---|
FP8 | 1.41x | 1.31x | 1.30x |
INT4 AWQ | 1.33x | 0.75x | 0.83x |
W4A8 AWQ | 1.38x | 1.00x | 1.15x |
Speedups are normalized to GPU count against the BF16 baseline. At batch 64, INT4 AWQ delivers 1,392.78 tokens per second against BF16's 1,679.74. That is 17.1% slower, from a format sold as compressing the model 4x to speed up inference.
The Architecture, Unpacked
Caption: Follow the dividing line in the second box. Three techniques need no training and four do, and the accuracy tables decide which side of that line you actually land on.
Three decisions carry this design.
One, effective bits as the cost axis. Not latency, not memory, a modeled average bit cost per eligible weight. It is cheap to compute, differentiable in the budget parameter, and lets you sweep a single number to trace the accuracy-versus-compression frontier. It is also a proxy, which the team names as a limitation: hardware-measured per-operator latency is on the roadmap because effective bits is not what you actually care about.
Two, coupling constraints live inside the search. TensorRT-LLM, vLLM and SGLang all require certain operator groups to share a format. AutoQuantize folds that into the ILP as a single decision with summed sensitivity and cost, rather than searching freely and repairing afterward. A repair pass would break the optimality guarantee the ILP just bought you.
Three, the diagonal approximation is load-bearing and acknowledged. Summing sensitivities across a QKV group assumes the Q, K and V quantization errors do not interact. The team says so and flags combinatorial effects as future work. That is the honest version of a shortcut that makes the whole thing tractable.
The Code, Annotated
The whole search is one call
import modelopt.torch.quantization as mtq
model, search_state = mtq.auto_quantize(
model,
constraints={"effective_bits": 4.8}, # the ONLY performance knob today
quantization_formats=[mtq.NVFP4_DEFAULT_CFG, mtq.FP8_DEFAULT_CFG],
data_loader=calib_loader,
forward_step=lambda model, batch: model(**batch),
loss_func=lambda output, batch: output.loss, # ← required, and this is why
num_calib_steps=512,
num_score_steps=128, # ← 4:1. Docs say keep 1:4 to 1:8.
)
# loss_func is not optional decoration. Sensitivity scoring needs a BACKWARD
# pass to get g_i = dL/dY_i, so auto_quantize needs a differentiable objective
# in a way that plain mtq.quantize() does not. Calibration-only PTQ never
# touches gradients; AutoQuantize does. That single line is the difference.
#
# num_score_steps < num_calib_steps because scoring is the expensive phase:
# it runs a backward pass AND replays every candidate format at every module.
Caption: The presence of loss_func tells you this is not really post-training quantization. It is a gradient-based search wearing PTQ's interface.
Effective bits is not the number you think
# ModelOpt's own accounting, from the AutoQuantize announcement:
FORMAT_BITS = {"NVFP4": 4.5, "FP8": 8, "BF16": 16}
# Why NVFP4 is 4.5 and not 4:
element_bits = 4 # the FP4 value itself
block_scale = 8 # one E4M3 scale factor...
block_size = 16 # ...shared across 16 elements
print(element_bits + block_scale / block_size) # 4.5 ← THIS is the trick
# So "FP4" carries a 12.5% metadata tax before a single layer is promoted.
# Nemotron 3 Ultra's searched sweet spot came out at 5.03 effective bits,
# only about 15% of the way from pure NVFP4 (4.5) toward pure FP8 (8).
# Read that as: a small number of promoted layers buys most of the accuracy.
#
# The announcement adds one more wrinkle worth knowing before you compare
# numbers: NVFP4 defaults measure ABOVE 4.5 because lm_head stays BF16.
Caption: Anyone benchmarking "FP4 versus FP8" on a bits axis without the 0.5 block-scale overhead is comparing the wrong two points.
The recovery step the API does not advertise
# The one-liner everyone quotes:
model = mtq.quantize(model, mtq.INT4_AWQ_CFG, forward_loop)
# Llama-2-7B validation loss on samsum, from examples/benchmark.md:
# BF16 baseline 1.036
# INT4 weights / FP16 activations 1.059 PTQ → +2.22% fine
# INT4 weights / INT8 activations 3.321 PTQ → +220.56% ← broken
# 1.294 QAT → +24.90%
#
# ← THIS is the trick, and it is a warning. Quantizing WEIGHTS to 4 bits
# costs 2.22% of validation loss. Adding 8-bit ACTIVATIONS costs 220.56%,
# a 99x larger hit from the cheaper-sounding change. Weights are static and
# well-behaved; activations carry per-token outliers that a single scale
# cannot cover. That asymmetry is the entire reason SmoothQuant exists.
import modelopt.torch.opt as mto
model = mtq.quantize(model, mtq.INT4_AWQ_CFG, forward_loop)
train(model, ...) # QAT recovers 88.7% of the damage
mto.save(model, "qat_checkpoint.pt") # state travels with the checkpoint
Caption: PTQ on weights alone is nearly free. PTQ on activations destroys the model and needs a training loop to undo. The library sells both under one word.
It In Action
Input: Qwen3.6-35B-A3B, four RTX 6000 Ada GPUs, 128 calibration samples at sequence length 512, searching over NVFP4 and FP8.
Step one, capture. For each scored module, ModelOpt records the BF16 output and replays the captured input through simulated quantization for every candidate format, giving the output error term.
Step two, gradient. One backward pass per scoring batch yields the gradient at each operator output. Squaring it approximates the Hessian diagonal via the diagonal Fisher.
Step three, score. Sensitivity is the squared-gradient-weighted output error, summed over the feature dimension. No explicit Hessian is ever formed.
Step four, group. Q, K and V collapse into one ILP decision with summed sensitivity. Gate and up projections collapse into another. MoE sparse experts collapse into one decision scored jointly at the block output.
Step five, solve. The ILP picks exactly one format per decision under the effective-bits budget.
The numbers. Gradient scoring takes about 16 minutes at 29 GB peak memory. KL-divergence scoring, which quantizes one layer at a time and re-evaluates the full model, takes about 14 hours at 23 GB. That is a 52.5x speedup for 26% more memory.
Downstream. The same machinery on Nemotron 3 Super completes mixed-precision PTQ in under two hours on one eight-GPU B200 node with 512 samples, reaching 99.8% median accuracy relative to BF16. On Nemotron 3 Ultra it landed on 5.03 effective bits with MoE routed experts in NVFP4, shared experts and Mamba projections in FP8, attention and latent MoE layers in BF16, and KV cache in FP8. One checkpoint that runs on both Hopper and Blackwell.
Why This Design Works, And What It Trades Away
It works because the sensitivity distribution across layers is genuinely lopsided. A handful of layers, attention projections and the network's final layers, are disproportionately fragile, while MoE experts are forgiving. Keeping the fragile few at higher precision costs little memory and recovers most of the accuracy, which is why 5.03 effective bits sits so close to the NVFP4 floor.
The second thing it gets right is refusing to search in a space the runtime cannot execute. Folding coupling constraints into the ILP means the returned assignment is deployable rather than aspirational.
What it trades away:
Effective bits is a proxy for the thing you care about. It models memory, not latency. The batch-size inversion in the benchmark table is precisely a case where fewer bits does not mean faster, and a search optimizing effective bits cannot see that. NVIDIA lists hardware-measured cost as the next step, which is the correct fix and an admission that the current objective is incomplete.
Isolated scoring ignores interactions. Each layer is scored quantized alone. Real quantization errors compound. The team flags this as future work.
Vendor benchmarks with vendor caveats. The PTQ table is v0.21.1 with TensorRT-LLM v0.15 at 2048 input and 128 output tokens, and the file itself says these should not be read as peak performance. Your prompt-heavy or long-output workload will move these numbers.
The support matrix is a maze. Eight separate support matrices across LLM, diffusers, ONNX, Windows, QAT, pruning, distillation and speculative decoding. Composing techniques means checking your model against several of them.
Everything good needs data. Calibration wants 128 to 512 samples. QAT, pruning recovery, distillation and sparsity recovery all want a training set that resembles your workload. The one-line API implies otherwise.
Technical Moats
The algorithms are published and the code is Apache 2.0. AWQ, SmoothQuant, SparseGPT, Minitron, Medusa and EAGLE are all in the literature. Reimplementing any single one is a known quantity.
The moat is the export matrix and the hardware coupling underneath it. A quantized checkpoint is only useful if a runtime can execute the exact format on the exact silicon. NVFP4 needs Blackwell tensor cores. FP8 needs Hopper or newer. The coupling rules AutoQuantize encodes come from TensorRT-LLM, vLLM and SGLang kernel constraints that change per release. Keeping one library's search space aligned with three moving runtimes across two GPU generations is continuous integration work that nobody outside the vendor can sustain.
The second moat is the Nemotron flywheel. Model Optimizer is how NVIDIA ships its own models, so every technique gets exercised on a 550B production checkpoint before it reaches you. The Nemotron 3 Ultra blog is a release note and a validation run at once.
The third is distribution. Preinstalled in the NGC PyTorch, NeMo and TensorRT-LLM containers, integrated with Megatron-Bridge, Megatron-LM and Hugging Face Accelerate. Being the default is worth more than being the best.
Insights
Insight One: four-bit quantization gets slower as batch size grows, and NVIDIA published the table.
Trace both formats across batch size on the H200 numbers in the repo.
Format | Llama3.1-8B batch 1 | batch 8 | batch 64 |
|---|---|---|---|
FP8 | 1.41x | 1.31x | 1.30x |
INT4 AWQ | 1.33x | 0.75x | 0.83x |
And on the 70B, where FP8 goes the other way entirely:
Format | batch 1 | batch 8 | batch 64 |
|---|---|---|---|
FP8 | 1.90x | 1.99x | 2.10x |
INT4 AWQ | 1.93x | 1.03x | 0.88x |
FP8 holds or improves as batch grows. INT4 AWQ falls below parity on both models. The mechanism is standard once stated: at batch 1 decode is memory-bandwidth-bound, so smaller weights win. As batch grows the workload becomes compute-bound, and INT4 has no native tensor-core path, so every matmul pays a dequantization tax that the bandwidth saving no longer offsets. FP8 has native hardware support and keeps its gain.
Now add accuracy. INT4 AWQ costs 5.66% MMLU on the 8B against FP8's 1.50%, which is 3.77x worse. On the 70B it is 1.07% against 0.38%.
So on an H200 at any serving batch size, INT4 AWQ is dominated by FP8 on both throughput and accuracy simultaneously. The only surviving reason to choose it is fitting a model into memory you do not have. That is a real reason. It is not the reason the technique is usually sold with, and the repo's own table has said so since v0.21.1.
Insight Two: the word "post-training" is doing work the tables do not support.
Three techniques, three accuracy tables, one pattern.
Quantization: INT4 weights with FP16 activations degrades validation loss 2.22% under PTQ. Add INT8 activations and it degrades 220.56%, a 99x larger hit. QAT pulls that back to 24.90%, recovering 88.7% of the damage.
Sparsity: SparseGPT without fine-tuning takes Llama-2-70B from 0.721 to 2.724 on Open-Orca, 277.8% worse. With fine-tuning, 1.01, recovering 85.6%. The docs recommend fine-tuning, which is a polite way of saying the training-free path is not shippable.
Pruning: the Minitron workflow is pruning plus distillation. Distillation is not optional polish, it is how the pruned model recovers.
The pattern holds across all three: the calibration-only version of each technique is a starting point, and the recovery step is where the accuracy lives. Which reframes what the library is. The one-line mtq.quantize() call is the marketing surface. The actual product is the training infrastructure underneath, the Megatron-Bridge and Megatron-LM and Accelerate integrations that let you run the recovery step at scale.
Budget accordingly. If your plan is a calibration script and an afternoon, that works for FP8 weight-and-activation on a large model and for weight-only 4-bit. For anything more aggressive, you are running a training job, and the library is honest about this in its tables while its README is not.
Takeaway
AutoQuantize's sensitivity scoring takes about 16 minutes where the KL-divergence method takes about 14 hours, on the same model and hardware. A 52.5x speedup, bought with 26% more peak memory, by changing the complexity class rather than optimizing anything.
The measured comparison is Qwen3.6-35B-A3B on four RTX 6000 Ada GPUs, 128 samples at sequence length 512. Gradient scoring: 16 minutes, 29 GB. KL divergence: 14 hours, 23 GB.
The mechanism is worth understanding because it generalizes. KL scoring answers the question exactly: quantize layer i in format f, run the full model, measure how the output distribution moved. That is one full-model pass per layer per format, so it scales as O(N² x F).
Gradient scoring refuses to answer exactly. It expands the loss to second order around each layer's output, notes that the first-order term vanishes in expectation for a trained model, discards everything off the Hessian diagonal, and estimates that diagonal with squared gradients. What remains is a gradient-squared-weighted output error needing one backward pass per batch plus a local replay per format. O(N x F).
The surprising part is not the speedup. It is that the estimate is good enough to beat exact measurement in practice, because the exact method was so expensive nobody swept the budget with it. Fourteen hours per sweep means you run one. Sixteen minutes means you run twenty and pick the knee of the curve. An approximation that unlocks search beats an exact measurement that forbids it.
That is the transferable lesson, and it is not really about quantization.
TL;DR For Engineers
INT4 AWQ runs Llama3.1-8B at 0.83x of BF16 throughput at batch 64 on an H200, and costs 3.77x more MMLU than FP8. Both numbers are in the repo's own benchmark file.
FP8 is the default answer on Hopper and later. It holds 1.30x on the 8B and improves to 2.10x on the 70B as batch grows, where 4-bit collapses.
Quantizing weights to 4 bits costs 2.22% of validation loss. Adding 8-bit activations costs 220.56%. Activations are the problem, which is why SmoothQuant exists.
Every technique's calibration-only path underperforms badly at aggressive settings. QAT recovers 88.7% of quantization damage, fine-tuning recovers 85.6% of sparsity damage.
NVFP4 is 4.5 effective bits, not 4. The E4M3 block scale adds 8 bits per 16 elements before anything is promoted.
Explain It Like I'm New
A trained model is a very large pile of numbers. Storing each one precisely is expensive, and moving them between memory and the processor is slower than the arithmetic itself. So the obvious saving is to store them less precisely: instead of sixteen bits per number, use eight, or four.
This works better than it has any right to, because trained models carry a lot of redundancy. But the redundancy is not spread evenly. A few parts of the network are fragile and fall apart when rounded aggressively, while most are unbothered.
The hard question is which parts. Traditionally you found out by trial and error, one experiment per layer, which took so long that most teams gave up and applied the same precision everywhere.
NVIDIA's Model Optimizer includes a search that answers this cheaply. Instead of testing each layer by running the whole model, it uses calculus to estimate how much each layer matters, then solves a packing problem: given an average precision you can afford, which layers get the expensive treatment.
The part worth carrying is a caution. Fewer bits does not automatically mean faster. When a system is waiting on memory, smaller numbers help. When it is busy computing, unpacking those smaller numbers costs time the compression does not repay, and the compressed model can end up slower than the original.
Compression is a trade whose sign depends on what your machine is waiting for.
See It In Action
The AutoQuantize announcement (nvidia.github.io), the full derivation from Taylor expansion to ILP, the grouped-decision rules, and Table 1 with the scoring cost comparison. The most technically substantial page NVIDIA has published on this library.
examples/benchmark.md (repo), the PTQ, QAT, sparsity and diffusion tables this issue is built on. Ten minutes here will change which format you deploy.
The Nemotron-3-Nano end-to-end tutorial (repo), pruning plus two-phase distillation plus FP8 reaching 2.6x vLLM throughput and 2.6x memory reduction. The only place all three techniques compose in one runnable workflow.
Creating the Nemotron 3 Ultra NVFP4 checkpoint (NVIDIA blog), AutoQuantize on a 550B production model, including the four-over-six block scaling that raises the global weight scale 1.75x and cuts median MSE 16.4%.
Puzzletron (repo), heterogeneous pruning and NAS where different blocks get different architectures. The most interesting thing in the library nobody is writing about yet.
Community Conversation
The December rename is the strategic signal. Dropping "TensorRT" from the name while adding first-class vLLM and SGLang export says NVIDIA decided the checkpoint format matters more than the runtime lock-in. Worth watching whether the coupling constraints stay runtime-neutral as they evolve.
The public roadmap (issue 1699) is an unusual artifact for a vendor library. The AutoQuantize post's "next steps" naming hardware-aware cost and combinatorial effects is a lab telling you where its current method is weakest.
Customer stories are doing the benchmarking NVIDIA is not. Bielik.AI's 33% smaller and 50% faster at 90% quality, and Domyn's 355B to 260B, are third-party numbers on real deployments. The Bielik post in particular is the closest thing to an independent Minitron reproduction.
The announcements moved to GitHub Pages in August, ahead of the AutoQuantize post. Publishing algorithm derivations in docs rather than a developer blog changes who the audience is, and this one has the equations in it.
The unanswered question is a latency-aware objective. Effective bits cannot express the batch-size inversion in NVIDIA's own benchmark table, so today the search can hand you an assignment that is smaller and slower. NVIDIA has said hardware-measured cost is coming. Until it lands, sweep the budget and benchmark the ends.
References
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, Xiao et al., ICML 2023, migrates activation outliers into weights, the fix for the 99x asymmetry above
Compact Language Models via Pruning and Knowledge Distillation, Muralidharan et al., NeurIPS 2024, the Minitron recipe behind ModelOpt's pruning workflow
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, Lin et al., MLSys 2024 best paper, the INT4 method in the benchmark table
Optimal Brain Surgeon, Hassibi and Stork, NeurIPS 1992, the second-order sensitivity idea AutoQuantize revives
SqueezeLLM: Dense-and-Sparse Quantization, Kim et al., ICML 2024, source of the diagonal-Fisher approximation
LLM-MQ: Mixed-Precision Quantization for Efficient LLM Deployment, Li et al., NeurIPS ENLSP 2023, the ILP formulation AutoQuantize builds on
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, Frantar and Alistarh, ICML 2023, the sparsity method whose recovery table appears above
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, Frantar et al., ICLR 2023, the other half of the PTQ canon
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, Cai et al., 2024, the speculative decoding path in the library
Summary
NVIDIA Model Optimizer bundles quantization, pruning, distillation, speculative decoding and sparsity behind one Apache 2.0 API, exporting to TensorRT-LLM, vLLM and SGLang, with AutoQuantize solving per-layer format assignment as an ILP under an effective-bits budget using gradient-based sensitivity scoring that runs 52.5x faster than KL divergence. Its own benchmark file shows INT4 AWQ running 17% slower than BF16 at batch 64 on an H200 while costing 3.77x more MMLU than FP8, and its accuracy tables show every technique's calibration-only path needing a training loop to recover. It matters because the library's real product is the recovery infrastructure, not the one-line quantize call.
Keep Going
The habit from this issue: when a vendor ships a benchmark file in its own repo, read it before reading the README. Two columns of NVIDIA's PTQ table invert the pitch on the page above them, and the whole finding is a matter of reading left to right.
SnackOnAI runs this teardown weekly on the systems engineers actually deploy, inference stacks, quantization pipelines, kernels, and the benchmark rows vendors do not put in the announcement. No announcements, no press release summaries. Subscribe at snackonai.com and join 10,000+ engineers reading it.
Forward this to whoever on your team is about to ship a 4-bit checkpoint to a high-throughput endpoint.
Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad, it helps us keep building and delivering value 🚀
How AI-Era Pricing Is Reshaping Finance Operations
Usage-based and hybrid pricing models are changing how B2B companies generate revenue — and creating new headaches for the finance teams behind them.
Tabs co-founder Rebecca Schwartz and PwC Partner Amit Dhir sat down to unpack exactly what that means in practice: how pricing model decisions ripple into revenue recognition, forecasting, and financial ops — and what it takes to scale without piling on manual work.
Watch the on-demand recording to get practical frameworks, real-world examples, and a clear path to operationalizing usage-based revenue — including a forward-looking take on how AI will reshape financial workflows. If your team is navigating pricing complexity heading into the back half of the year, this is worth an hour.