MaverIQ Shows Fragmented GPU Memory Is Capacity You Already Paid For

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 30, 2026

The promise: by the end of this issue you will understand how MaverIQ predicts an LLM's latency and memory from a one or two layer "fingerprint," why uneven layer placement barely costs anything at inference time, and which of its headline numbers hold up when you rerun its estimator on its own data.

The fence: we cover the MaverIQ repository (last commit November 16, 2025: controller, estimators, profiling CSVs, the artifact notebook), the SC'25 paper MaverIQ: Fingerprint-Guided Extrapolation and Fragmentation-Aware Layering for Intent-Based LLM Serving by Liakopoulos, Sinha, Hu, Lee and Yadwadkar, and TensorRT-LLM, which it modifies. We did not rerun the GPU experiments (they need eight RTX A6000s and a patched TensorRT-LLM 0.9 build). We did replay the shipped latency estimator, layer distributor and uneven-layering measurements against the CSVs the authors published. Everything else is project reported.

MaverIQ was recently presented at AMD AI DevDay Austin on October 29, 2026, by Neeraja J. Yadwadkar, Assistant Professor and Fellow of the AMD Chair in Computer Engineering at UT Austin, in her talk “Turning User Goals into Efficient AI Deployments,” which also previewed upcoming work with AMD to extend the approach across diverse models, hardware and user goals.

MaverIQ is an inference serving controller. A user says what they care about, an intent such as "minimize cost," "minimize latency," "stay under this latency SLO" or "keep at least this much accuracy." MaverIQ picks a deployment configuration and places it on a shared GPU cluster.

The configuration space is bigger than it looks. Llama-2-70B alone has about 180 configurations on an 8-GPU node, combining tensor parallelism (TP, splitting each layer across GPUs), pipeline parallelism (PP, splitting the stack of layers into stages), quantization (FP16, INT8, INT4, GPTQ) and an optional INT8 KV cache. Across them, latency varies about 8x and user cost about 25x.

Two ideas carry the system. Fingerprints make choosing a configuration cheap. Fragmentation-aware layering makes placing it cheap.

Caption: The expensive steps happen once per model on a tiny proxy. Every deployment decision after that is arithmetic plus a memory scan of the live cluster.

Four design decisions matter most.

One: profile a slice, not the model. A transformer is the same block repeated N times, so latency and memory are close to linear in layer count. A fingerprint keeps the embedding, final norm and output head but only one or two hidden layers. For Llama-2-70B it is 41x smaller and 63x faster to profile.

Two: model parallelism analytically. Instead of measuring every TP and PP combination, MaverIQ measures three reference points, L(1,1), L(1,2) and L(2,1), and fits four exponents (α, β, γ, δ) once with gradient descent to capture how communication overhead scales.

Three: let intent pick the objective. The default min_cost objective is latency multiplied by memory, effectively GB-seconds. Accuracy floors come from a per-configuration MMLU table relative to FP16.

Four: fill the gaps unevenly. The placer sorts GPUs by free memory and hands out layers one at a time to any stage that still has room. A model too big for any balanced split can still land on GPUs other tenants have partly filled.

First, the fingerprint extrapolation, verbatim from tools/controller/MS_latency_estimator.py. Four measurements (1 and 2 layers, 10 and 20 output tokens) solve for four unknowns.

Caption: Measure a two-layer toy, multiply the per-layer slope by the real layer count. Cheap and elegant, but any error in a millisecond-scale slope is multiplied by up to 80.

Second, the parallelism model, verbatim from the same file.

Caption: Three terms: ideal speedup, tensor-parallel all-reduce cost, pipeline hand-off cost. Three reference measurements plus four fitted exponents cover all 180 configurations.

Third, the fragmentation-aware placer, from tools/controller/MS_controller.py (trimmed).

Caption: No optimizer, no ILP. A greedy round-robin that stops giving layers to full GPUs is enough, because uneven stages cost almost no latency.

Fourth, our replay of the repo's own data.

Caption: The placement claim reproduces cleanly. The estimation claim reproduces exactly on one profiling CSV and degrades sharply on the other.

Input: u1 deploy llama-2-70b 100 min_cost on an 8x A6000 node where other models already use 8, 12, 18, 24, 32, 36, 40 and 44 GB of each 48 GB GPU. That leaves 170 GB free in total, but no single GPU has more than 40 GB.

Step one, estimate. MaverIQ evaluates every TP, PP, dtype and KV option from the stored fingerprint profile. FP16 Llama-2-70B needs about 137 GB at PP=8 in the project's measurements, roughly 1.71 GB per layer.

Step two, rank. For min_cost, each feasible configuration is scored by estimated latency times memory. Suppose FP16 with TP=1 and PP=8 wins (the accuracy floor rules out compression).

Step three, place. A balanced split needs 10 layers, about 17.1 GB, on every GPU. Four of the eight GPUs have only 16, 12, 8 and 4 GB free, so a balanced deployment would need fresh GPUs. Running the repo's distribute_layers on these numbers gives the map [15, 15, 14, 14, 9, 7, 4, 2], all 80 layers on GPUs that are already partly busy.

Step four, build and serve. The patched TensorRT-LLM builds an engine with that --pp_map, and requests join the user's FCFS queue.

Output: the closest measured layout in the repo (2 to 18 layers per GPU) produced 100 tokens in 20.325 s, against 20.302 s for the balanced layout: a 0.11% penalty for using capacity that would otherwise sit stranded.

Caption: Measured by the authors, from unbalanced_PP_data_collection.csv. Our worked scenario's tenant memory is illustrative; the layer map comes from running the shipped placer on it.

Why it works. A decoder-only LLM is a stack of identical blocks, so per-layer costs are nearly constant and a two-layer slice predicts the whole. Uneven pipelines are cheap at inference for a structural reason: without microbatch pipelining, a request passes through the stages one after another, so total time is the sum of all layers' work wherever they sit. Imbalance creates the idle "bubbles" that hurt training, where many microbatches overlap; a forward-only serving pass has far fewer to lose. Once placement no longer needs balance, every free gigabyte on any GPU becomes usable.

What it trades away.

Insight One: fingerprint profiling is accurate on average and fragile in practice. We replayed the shipped estimator against the repo's 2,060 full-model measurements. With the batch-size-1 fingerprint file the notebook uses, we reproduced the paper's median error exactly: 4.39% at 100 output tokens. The runtime controller and standalone estimator load a different file, fingerprint_data_collection_BS.csv, whose batch-size-1 rows profile the same fingerprints with latencies a median 24% lower. With that file, median error rises to 11.8% at 100 tokens, 19.3% at 50 tokens and 53.1% at 10 tokens. Both are legitimate measurement runs; the point is that a two-layer probe timed in tens of milliseconds amplifies run-to-run noise 30 to 80 fold. Anyone adopting the technique should profile each fingerprint several times and take the median.

Insight Two: the most dramatic serving wins are baselines falling over. The artifact notebook reports latency reductions of 99.5% against AlpaServe in high-load runs, and a median latency of 0.33 s for MaverIQ against 76.97 s for AlpaServe* and 502.77 s for Hugging Face Accelerate in one setting. Those are queueing collapses, not efficiency gains. The cleaner comparison is the FP16-only run: 28% lower latency and 3.8x lower provider cost against the strongest baseline, AlpaServe*. That is still a large win, and it comes mostly from placing more models on fewer GPUs.

At inference time, pipeline balance barely matters. Across the authors' measurements of Llama-2 7B, 13B and 70B on eight GPUs, the worst layer map cost at most 0.50% latency at batch sizes from 1 to 64, including Llama-2-70B split as 2 to 18 layers per GPU (+0.11%). The balanced-pipeline rule is inherited from training, where overlapping microbatches make the slowest stage everyone's bottleneck. In serving it mostly strands memory: in our replay, four of eight partly used GPUs could not host a balanced 70B split, but the greedy uneven placer fit all 80 layers.

Running a large AI model means splitting it across several graphics cards, because no single card has enough memory. The usual approach is to cut the model into equal pieces, one per card, like dividing a long train into identical carriages.

The trouble is that in a shared data center, cards are rarely empty. One has 40 gigabytes free, another 4, another 16. Equal pieces only fit where there is enough room for every piece, so much of that leftover space goes unused, and providers buy more cards instead.

MaverIQ asks a simple question: do the pieces really need to be equal? For training, yes, because the cards work in parallel and the slowest one holds everyone up. For answering questions, a request mostly moves through the cards one after another, so it barely matters which card does more of the work. Big pieces go where there is lots of room, tiny pieces into the gaps.

It also avoids testing every possible setup by timing a miniature version of the model, one or two of its repeated building blocks, and scaling the numbers up, like estimating a skyscraper's elevator time from one floor.

The bigger lesson is that much AI cost is wasted space, not missing hardware. Smarter placement can unlock capacity that companies have already paid for.

MaverIQ makes two bets, and only one is load-bearing. The fingerprint profiler is clever and much cheaper than sweeping configurations, but it is sensitive to timing noise in ways a single median hides. The uneven-layering insight is rock solid: measured across three model sizes and five batch sizes, imbalanced pipeline stages cost under half a percent of latency, and that frees every stranded gigabyte in a shared cluster. If you operate multi-tenant GPU inference, steal the placer first. Then profile your fingerprints more than once.

MaverIQ chooses LLM deployment configurations from user intents using one and two layer fingerprints plus an analytical parallelism model, then places models unevenly across fragmented GPUs. Its uneven-layering result is robust (≤0.5% latency cost on Llama-2 models), while its fingerprint estimates are cheaper than full profiling but sensitive to measurement noise, especially for short outputs.

A shopper types “running shos” and gets nothing back. Another types “red waterprof trail runners under $150” and hits the same wall. Both knew what they wanted.

Traditional site search relies on exact keyword matching, leaving shoppers stranded at “0 results found” screens.

Gladly understands what shoppers mean, typos and all. Instead of a dead end, they get products that fit what they asked for and a conversation that keeps them shopping.

See how Gladly turns dead-end searches into sales

Turn Dead Ends Into Sales