In partnership with

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | September 11, 2026

The Promise

llmfit publishes the formula behind every tokens-per-second number it prints, which means you can check it, and checking it reveals two estimators running different physics under one column heading.

What this covers: the roofline speed model and its fallback, why the GPU bandwidth table is load-bearing, the fit-band arithmetic, and what the confidence field is actually for. What this excludes: the TUI, the web dashboard, and the download and serve integrations.

What It Actually Does

llmfit is a Rust CLI from Alex Jones that reads your hardware and tells you which open models will run well on it. MIT licensed, on crates.io, Homebrew, Scoop, MacPorts and uv, with Authenticode-signed Windows binaries through the SignPath Foundation. It is currently trending on Trendshift.

The pitch is that it saves you from downloading forty gigabytes to discover a model does not fit. That framing undersells it.

What llmfit is really doing is publishing a performance model. Hardware detection through nvidia-smi, rocm-smi, npu-smi, sysfs and system_profiler. A model catalog scraped from the Hugging Face API and baked into the binary with include_str!. Then four scores per model, zero to a hundred: quality, speed, fit and context, combined with weights that shift by use case, so Chat weights speed at 0.35 while Reasoning weights quality at 0.55.

The part worth your attention is section five of its own docs, where it writes down the formula:

tokens_per_second = (bandwidth_GB_s / model_size_GB) × 0.55

That is a roofline model. Each decoded token requires reading every weight once, so throughput is memory bandwidth divided by the bytes you must read, discounted for kernel overhead and KV-cache traffic.

Check it. An RTX 4090 has 1,008 GB/s. Llama-3-8B at Q4_K_M is about 4.7 GB. That gives 118 tokens per second, and published llama.cpp figures for that combination land between 120 and 140. The constant is defensible and slightly conservative, and the Advanced Configuration popup lets you tune it.

The Architecture, Unpacked

Caption: Focus on the two speed paths and the loop at the bottom. The paths decide how wrong a number can be, and the loop is how wrong numbers get replaced.

Three decisions carry this design.

One, the estimate ships its own inputs. llmfit info reports estimate_basis.gpu_bandwidth_gbps, so whichever bandwidth figure produced a number is visible, whether it came from an override, the table, or nowhere. Almost no tool in this category does that. It is what makes the rest of this issue possible to write.

Two, degrade context before degrading weights. When nothing fits at full context, the walk retries at half context rather than dropping to a harsher quantization. That is an opinionated call, and the right one for most local use, because a Q2_K model that fits is usually worse than a Q4 model with a shorter window.

Three, the verdict is one ratio, then a cap. Utilization decides the band, execution path caps it. No compound heuristic, which matters because of what happened the last time there was one.

The Code, Annotated

The formula, and where it diverges

EFFICIENCY = 0.55          # kernel overhead, KV-cache reads, memory controller effects
                           # tunable in the TUI via the Advanced Configuration popup

def decode_tps_roofline(bandwidth_gb_s, model_size_gb):
    """PATH A. Every decoded token reads every weight once, so throughput is
    bandwidth divided by bytes. quantization shrinks model_size_gb, so it
    moves this number directly and correctly."""
    return bandwidth_gb_s / model_size_gb * EFFICIENCY

BACKEND_K = {"CUDA": 220, "Metal": 160, "ROCm": 180, "SYCL": 100,
             "CPU_ARM": 90, "CPU_X86": 70, "NPU_ASCEND": 390}

def decode_tps_fallback(backend, params_b, quant_speed_multiplier):
    """PATH B, for a GPU not in the ~80 entry bandwidth table.
    ← THIS is the trick, and the trap. The denominator is PARAMS, not BYTES.
    quantization can only enter through a multiplier, so this is a curve fit
    rather than a roofline. The two paths are not the same model of reality."""
    return BACKEND_K[backend] / params_b * quant_speed_multiplier

# Llama-3-8B at Q4_K_M is 4.7 GB and 8.0B params. Same model, two paths:
#   RTX 4090, recognized (1008 GB/s):  1008 / 4.7 * 0.55 = 118.0 tok/s
#   same card, NOT recognized:           220 / 8.0 * mult =  27.5 tok/s
# A 4.3x gap on identical hardware, decided entirely by a table lookup.

Caption: Two functions, two physical models, one column in the output. Whether your GPU is in the table decides which one you get, and the error is a multiple rather than a percentage.

Why prefill is a different question entirely

def prefill_tps(active_params, gpu_tflops_fp16):
    """Prompt processing is COMPUTE bound: roughly 2 x active_parameters
    FLOPs per prompt token. Memory bandwidth says nothing useful here."""
    if gpu_tflops_fp16 is None:
        return None        # ← NOT 0.0. Absence is not zero, and 0.0 would
                           # read as "immeasurably slow" rather than
                           # "not estimated". A type decision, not a style one.
    return gpu_tflops_fp16 * 1e12 / (2 * active_params)

# quantization is ABSENT from this function on purpose. llama.cpp and vLLM
# both dequantize to fp16 for the matmul, so the FLOP count never changes.
#
# Llama-3-8B on an RTX 4090 at roughly 80 effective fp16 TFLOPS:
#   prompt   4,096 tok -> prefill 0.82 s | decode 500 tok 4.24 s | prefill 16% of wall time
#   prompt  16,384 tok -> prefill 3.28 s | decode 500 tok 4.24 s | prefill 44%
#   prompt  32,768 tok -> prefill 6.55 s | decode 500 tok 4.24 s | prefill 61%
# Past ~20k tokens of prompt, quantization is optimizing the smaller half.

Caption: The absence of a quantization term is the most informative thing in this function. For RAG and long-code workloads, the compression you chose barely touches your latency.

The fit band, and the heuristic it replaced

def fit_verdict(memory_required_gb, memory_available_gb, run_mode):
    u = memory_required_gb / memory_available_gb
    band = ("Perfect" if u <= 0.60 else
            "Good"    if u <= 0.85 else
            "Marginal" if u <= 0.98 else   # ← 98, not 100. The last 2% is
            "Too Tight")                    # allocator slack and fragmentation.
                                            # A pool filled to 100% does not load.
    if run_mode in ("MoE_OFFLOAD", "CPU_GPU", "CPU"):
        band = min(band, "Good", key=ORDER)  # Perfect means roomy AND on the GPU
    return band

# The heuristic this replaced, from the project's own docs:
#   recommended_ram_gb = model_size * 2.0
# It broke in both directions at once:
#   23 GB model on a 24 GB card met its 22 GB recommendation -> "Perfect"
#     actual utilization 96%, and it will not load
#   9 GB model at 56% of a 16 GB card -> only "Good"
#     same model on a 24 GB card -> "Perfect"
#     the verdict was tracking the CARD's size, not how tightly the model fit

Caption: A compound heuristic that over-promised on tight fits and under-rated roomy ones simultaneously. Replacing it with a single ratio fixed both, which is the argument for one number over two.

It In Action

Input: an RTX 4090 with 24 GB VRAM, asking for a coding model.

Step one, detect. nvidia-smi reports 24 GB, backend resolves to CUDA, and the bandwidth table matches the card at 1,008 GB/s.

Step two, walk the quantizations. For Llama-3-8B the walk starts at Q8_0, roughly 8.5 GB, which fits at 35% utilization. Q8_0 wins on quality and the walk stops.

Step three, score. Utilization 35% is under 60%, so the verdict is Perfect, uncapped because the run mode is GPU.

Step four, estimate speed. 1,008 divided by 8.5 times 0.55 gives 65.2 tokens per second at Q8_0. Had memory forced Q4_K_M at 4.7 GB, the same formula gives 118.0.

Step five, tag the confidence. No local benchmark, no community match on this exact configuration, so the field reads estimated. The number prints identically to a measured one. Only the tag distinguishes them.

Step six, verify. llmfit bench measures real tokens per second and time to first token against your running provider. That result becomes measured_local and overrides the estimate in your own fit table. llmfit bench --share opens a PR from inside the TUI, and once merged it ships in the next release as measured_community for anyone on matching hardware.

The numbers that matter. Q8_0 at 65.2 against Q4_K_M at 118.0 is an 81% decode speedup for a quantization step. On a 32k-token prompt, that same step changes prefill by exactly nothing, because prefill costs 6.55 seconds either way.

Why This Design Works, And What It Trades Away

It works because it refuses to hide. The formula is in the docs, the constants are in a table, the bandwidth figure used is echoed back in estimate_basis, and the confidence field tells you whether anyone measured anything. Most hardware-advisor tools print a number and a vibe. This one prints a number and its derivation.

The MoE handling is the other genuinely good piece. Sparse models are estimated from active parameters, with a tier one path that decomposes per-token traffic into expert FFN weights that scale with quantization and attention, router and embedding weights that do not. Mixtral 8x7B has 46.7B total parameters but activates about 12.9B, which takes VRAM from 23.9 GB to roughly 6.6 GB with expert offloading. The named alternative, llm-checker, treats all models as dense and therefore gets Mixtral and DeepSeek-V3 badly wrong.

What it trades away:

The bandwidth table is a cliff, not a slope. Insight One.

The catalog is frozen at compile time. include_str! means model updates arrive by upgrading the binary. That is a real tradeoff for correctness and reproducibility, and it means a model released this week is invisible until the next release.

Quality scoring is partly reputational. The quality dimension combines parameter count, family reputation, quantization penalty and task alignment from a curated benchmark table, with name-based heuristics for families without an entry. Those heuristics are the softest thing in the system, and the docs say corrections are welcome pull requests.

The MoE tier two fallback has a disclosed 2.6x error. Charging architectures with 128 or more experts as heavy router overhead puts gpt-oss-120b about 2.6 times low. The maintainer names the model and the factor, which is more than most projects would do, and a 2.6x error on tokens per second is still the difference between usable and not.

Nothing here models the thing you will actually hit first. Estimates assume the model is loaded and running. They say nothing about load time, disk throughput, or the memory spike during weight loading, which is where a lot of first attempts actually fail.

Technical Moats

There is no algorithmic moat. A roofline estimator is a division, the fit bands are four comparisons, and the whole thing is MIT licensed Rust you could fork this afternoon.

The moat is the calibration data, and it is being deliberately farmed. The bandwidth table covers roughly 80 GPUs, each entry a real specification someone looked up. The use-case benchmark table is aggregated from public leaderboards by hand. The MoE per-architecture efficiency and overhead pairs exist because someone noticed gpt-oss-120b was 2.6x off and fixed that one architecture without disturbing the others. None of that is research. All of it is accumulated, unglamorous correction.

The second moat is the submission path. llmfit bench --share opens a pull request from inside the terminal, with no gh CLI and no third-party account. Every friction point removed from that flow raises the measurement rate, and the measurement rate is the entire asset. A competing tool with a better formula and no contribution loop loses over time to a worse formula that keeps getting corrected.

Third, distribution breadth for a tool this young: Homebrew, MacPorts, Scoop, crates.io, uv, Docker, and signed Windows binaries. Plus a sister ecosystem in llmserve, llama-panel and a third-party Windows GUI.

Insights

Insight One: whether your GPU is in a lookup table changes the estimate by a multiple, not a margin.

Two estimators, and the docs are clear about when each fires. Recognized GPUs get the roofline. Unrecognized ones get K / params_b × quant_speed_multiplier.

Run the same model, Llama-3-8B at Q4_K_M, through both:

GPU

Bandwidth

Roofline estimate

CUDA fallback

Ratio

RTX 4060

272 GB/s

31.8 tok/s

27.5 tok/s

1.16x

M2 Max

400 GB/s

46.8 tok/s

27.5 tok/s

1.70x

RTX 3090

936 GB/s

109.5 tok/s

27.5 tok/s

3.98x

RTX 4090

1,008 GB/s

118.0 tok/s

27.5 tok/s

4.29x

A100 80GB

2,039 GB/s

238.6 tok/s

27.5 tok/s

8.68x

H100 SXM

3,350 GB/s

392.0 tok/s

27.5 tok/s

14.26x

The fallback is a single constant divided by parameter count. It cannot know whether your CUDA device is a laptop 4060 or an H100, so it returns the same 27.5 tokens per second for both. On the low end that is nearly right. On the high end it is fourteen times low.

This is not a bug, and the docs point at the fix: gpu_bandwidth_gbps_override exists precisely so you can supply the missing number, and whatever value was used comes back in estimate_basis. But it reframes what the tool is. The bandwidth table is not a detail, it is the product, and the failure mode is silent because both paths print into the same column.

The practical rule: run llmfit doctor, check whether your card resolved a bandwidth figure, and if it did not, look up your card's specification and set the override before trusting anything. That is a two-minute step which changes the answer by up to an order of magnitude.

Insight Two: the estimate is not the product. The confidence tag is.

Look at what the confidence field actually encodes: measured_local, measured_community, calibrated, estimated, unsupported. Three of those five mean a human ran something. Only one means pure formula.

Now look at the loop. llmfit bench measures on your machine and overrides the estimate locally. llmfit bench --share opens a pull request from the TUI. Merged submissions ship in the next release, so the next person on identical hardware sees a measured figure before running anything.

That is not a calculator with a benchmarking feature attached. It is a distributed measurement program with a calculator as the acquisition funnel. The formula's job is to be useful enough that people install the tool, and wrong enough in interesting cases that they benchmark and submit.

Which reframes the roadmap. The interesting metric for this project is not estimate accuracy. It is what fraction of fit rows on a typical machine read measured_ rather than estimated, and how fast that fraction is climbing. That number is not published anywhere, and it is the one that says whether the design is working.

It also explains the compile-time catalog, which looks like a limitation and is partly a consequence. Measurements are pinned to a release, so a number you see is reproducible against a specific binary. A live remote index would break that.

Takeaway

quantizing a model from Q8_0 to Q4_K_M nearly doubles decode throughput, 65.2 to 118.0 tokens per second on an RTX 4090. It changes prompt processing by exactly zero, because llama.cpp and vLLM dequantize to fp16 for the matmul and the FLOP count never moves.

llmfit encodes this by omitting quantization from its prefill function entirely. The absence is the statement.

Work through what that means on real prompts, using an 8B model on a 4090 at roughly 80 effective fp16 TFLOPS:

Prompt length

Prefill

Decode of 500 tokens

Prefill share

4,096 tokens

0.82 s

4.24 s

16%

16,384 tokens

3.28 s

4.24 s

44%

32,768 tokens

6.55 s

4.24 s

61%

Past roughly 20,000 tokens of prompt, more than half your wall clock is in the phase quantization does not touch. For retrieval-augmented workloads, long code contexts and document analysis, the compression choice everyone agonizes over is optimizing the smaller half of the job.

The corollary is more useful than the observation. If your workload is long-prompt and short-answer, stop tuning quantization and start looking at compute throughput, prompt caching and context reuse. If it is short-prompt and long-answer, quantization is your main lever and the roofline model tells you exactly what it buys.

Two workloads, same model, opposite optimization strategies. The tool reports both numbers separately, and prints null rather than a fake zero when it cannot estimate one. That distinction is a small thing that most tools get wrong.

TL;DR For Engineers

  • Decode estimate is (bandwidth_GB_s / model_size_GB) × 0.55. On a 4090 with 8B at Q4 that gives 118 tok/s against published llama.cpp figures of 120 to 140.

  • Unrecognized GPUs fall back to K / params_b, which returns 27.5 tok/s for every CUDA device. That is 1.16x low on a 4060 and 14.26x low on an H100. Set gpu_bandwidth_gbps_override.

  • Prefill costs 2 × active_params FLOPs per prompt token and ignores quantization entirely, because both major runtimes dequantize to fp16. Past 20k tokens it is most of your latency.

  • Fit verdict is one ratio: 60, 85 and 98 percent of pool utilization. The band stops at 98 because allocator slack needs the last two points.

  • Every row carries a confidence tag. estimated and measured_local print the same way, and telling them apart is the whole point of the tool.

Explain It Like I'm New

Running an AI model on your own computer has an awkward first step. The files are enormous, often tens of gigabytes, and you generally find out whether your machine can handle one only after downloading it and watching it fail.

The obvious fix is a compatibility checker. Read the specifications of the machine, read the requirements of the model, compare. That is roughly what people expected.

llmfit does something more interesting, because the honest answer is not a yes or a no. It is a speed, and speed depends on physics you can calculate in advance.

Generating text is mostly an act of reading. To produce each word, the computer must read the entire model from memory. So the speed is set by how fast memory can be read, divided by how big the model is. Shrink the model, which is what compression does, and it reads faster.

But there is a second phase, and it obeys a different rule. Before answering, the machine must read your question, and that part is limited by raw calculation rather than memory speed. Compression does not help it at all.

Two phases, two bottlenecks, and the popular advice only addresses one of them.

The wider lesson is about tools that show their working. This one publishes its formulas, so you can check whether a number applies to you, rather than taking it on faith.

See It In Action

  • How llmfit works (docs/how-it-works.md), the formulas, the backend constants, the fit bands and the confidence hierarchy, all written down. Unusually complete for a tool this young, and the source for everything in this issue.

  • The benchmarking guide (docs/benchmarking.md), the download, serve, measure and submit loop, including the pull request from inside the TUI. Worth running once even if you never use the tool again, because it will tell you how wrong your assumed throughput was.

  • llmfit info "<model>", the single most useful command, because it shows what an estimate assumes and how to verify it. Run it before llmfit fit and you will read the table differently.

  • llm-checker (repo), the alternative llmfit names itself, which actually pulls and runs models through Ollama rather than estimating. Useful contrast, and its lack of MoE support shows exactly what the sparse-model handling buys.

  • The llama.cpp benchmark threads the estimator is validated against (Apple Silicon, NVIDIA T4), the community measurement baseline for local inference throughput, and a good sanity check on any tool's claims.

Community Conversation

  • The docs name their own failure modes with numbers. The gpt-oss-120b figure being 2.6x low under the expert-count fallback is stated in the reference documentation, not buried in an issue. Projects that publish their own error factors are rare enough to be worth supporting.

  • The removed recommended_ram_gb heuristic is documented as a postmortem inside the reference docs, including both directions in which it failed. That is a design-decision log where most projects would have shipped a silent commit.

  • Signed Windows binaries via the SignPath Foundation with maintainer-approved signing requests, on a tool at this stage, signals someone who expects enterprise use. The explicit privacy statement about no network traffic without user action points the same way.

  • The sister project cluster of llmserve, llama-panel and a community-built Windows GUI suggests a maintainer building a local-inference toolchain rather than a single utility, which is worth tracking if you are choosing something to depend on.

  • The open question worth asking the maintainer: what fraction of fit rows resolve to measured_ rather than estimated on a typical machine today. That single number tells you whether the community measurement loop is working, and nobody has published it.

References

llmfit is an MIT-licensed Rust CLI that detects your hardware and scores every model in a compile-time catalog across quality, speed, fit and context, publishing the roofline formula behind each tokens-per-second figure so you can check it. Doing that check reveals two estimators with different physics sharing one output column, where an unrecognized GPU can be estimated fourteen times too slow, and a prefill function that omits quantization entirely because both major runtimes dequantize to fp16. It matters because the confidence tag, not the estimate, is the real product, and the community measurement loop behind it is the asset nobody is measuring.

Keep Going

The habit from this issue: when a tool publishes its formula, run the formula against a case it was not advertised on. Two divisions on an H100 turned a documented fallback into a fourteen-fold error, and the fix is a two-minute config override.

SnackOnAI runs this teardown weekly on the systems engineers actually deploy, inference stacks, local runtimes, estimators, and the constants vendors bury in a reference doc. No announcements, no press release summaries. Subscribe at snackonai.com and join 10,000+ engineers reading it.

Forward this to whoever on your team keeps saying the quantized model will be faster.

Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad, it helps us keep building and delivering value 🚀

How Jennifer Aniston’s LolaVie brand grew sales 40% with CTV ads

For its first CTV campaign, Jennifer Aniston’s DTC haircare brand LolaVie had a few non-negotiables. The campaign had to be simple. It had to demonstrate measurable impact. And it had to be full-funnel.

LolaVie used Roku Ads Manager to test and optimize creatives — reaching millions of potential customers at all stages of their purchase journeys. Roku Ads Manager helped the brand convey LolaVie’s playful voice while helping drive omnichannel sales across both ecommerce and retail touchpoints.

The campaign included an Action Ad overlay that let viewers shop directly from their TVs by clicking OK on their Roku remote. This guided them to the website to buy LolaVie products.

Discover how Roku Ads Manager helped LolaVie drive big sales and customer growth with self-serve TV ads.

The DTC beauty category is crowded. To break through, Jennifer Aniston’s brand LolaVie, worked with Roku Ads Manager to easily set up, test, and optimize CTV ad creatives. The campaign helped drive a big lift in sales and customer growth, helping LolaVie break through in the crowded beauty category.

Learn more.

Recommended for you

View all
caret-right
© 2026 Snack On AI.
beehiivPowered by beehiiv