Sponsored by

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | August 25, 2026

The Promise

Rex-Omni turns object detection into vocabulary lookup, and the reinforcement stage everyone credits fixed the model's behavior, not its geometry. The distinction decides whether you can ship it.

What this covers: the coordinate token scheme, the data engine chain, what GRPO measurably changed, the token economics, and the production limits. What this excludes: the full benchmark sweep across all eleven tasks, keypoint OKS methodology, and fine-tuning mechanics.

What It Actually Does

Rex-Omni is a 3B multimodal model from IDEA Research, the lab behind Grounding DINO, T-Rex2 and DINO-X. The paper landed October 2025 and has since been accepted to CVPR 2026. Weights are on Hugging Face, with an AWQ build at half the storage.

The mechanism is smaller than the framing suggests. Take Qwen2.5-VL-3B-Instruct. Repurpose the last 1,000 token IDs in its existing vocabulary as coordinate symbols <0> through <999>, each a quantized relative position. Add no parameters. Emit boxes, points, polygons and keypoints as one autoregressive token stream.

A box becomes four tokens instead of fifteen atomic digit tokens. That is the whole architectural contribution.

The zero-shot COCO results, F1 rather than mAP because these models have no usable confidence scores:

Model

F1@IoU 0.5

F1@IoU 0.95

F1@mIoU

DINO-Swin-L (trained on COCO)

75.6

25.4

62.1

Rex-Omni

72.0

15.9

52.9

SEED1.5-VL

71.3

14.3

51.4

Grounding DINO-Swin-T

69.8

23.0

56.6

DINO-R50 (trained on COCO)

68.8

21.1

55.6

Qwen2.5-VL-3B

64.7

15.0

47.6

Read the columns, not the headline. At the loose threshold Rex-Omni beats a COCO-trained DINO-R50 and an open-set Grounding DINO without seeing a single COCO training image. At the strict threshold it loses to both, badly. On the mean across thresholds it loses to both again, 52.9 against 55.6 and 56.6.

The abstract's claim of performance comparable to or exceeding regression models is true at IoU 0.5 and false at IoU 0.95. Where it wins outright is anywhere language carries weight: LVIS long-tail F1@mIoU 46.9 against Grounding DINO's 38.8, DocLayNet layout grounding 89.5 against SEED1.5-VL's 54.9, RefSpatial placement 50.0 against Gemini-2.5-Pro's 24.2.

The Architecture, Unpacked

Caption: Focus on the two lines at the bottom of stage two. The reward functions are geometry-aware, and the geometry metric did not move. Everything GRPO bought is in the line above it.

Three decisions carry the design, ranked by how much they actually matter.

One, the vocabulary tail is repurposed rather than extended. No new embedding rows, no resized output head, no architectural fork from Qwen2.5-VL. The cost is 1,000 tokens of natural language capacity that Rex-Omni can no longer emit. The benefit is that the model stays a drop-in Qwen2.5-VL derivative, and every upstream improvement to that backbone remains inheritable.

Two, relative quantized coordinates instead of absolute digits. Absolute coordinates make the classification target unbounded and image-size dependent. Bounding it to 1,000 bins turns localization into a fixed 1,000-way problem the model can actually learn from finite data.

Three, one output grammar for every task. Detection, pointing, OCR polygons, GUI grounding and keypoint JSON all serialize through <|object_ref_start|>PHRASE<|object_ref_end|><|box_start|>COORDS<|box_end|>. Boxes sort by x0 ascending, which gives the autoregressive decoder a stable ordering to learn rather than an arbitrary permutation.

The Code, Annotated

The decoding config is a confession

from PIL import Image
from rex_omni import RexOmniWrapper, RexOmniVisualize

rex = RexOmniWrapper(
    model_path="IDEA-Research/Rex-Omni",
    backend="transformers",          # "vllm" for throughput; batching matters
                                     # more here than for a one-shot detector
    max_tokens=2048,
    temperature=0.0,                 # deterministic
    top_k=1,                         # ...and greedy. Sampling is fully disabled.
    top_p=0.05,
    repetition_penalty=1.05,         # ← THIS is the tell
)
# Under temp=0 and top_k=1 decoding is already argmax, so top_p is inert and
# repetition_penalty is the ONLY knob still doing work. Shipping a repetition
# penalty on a detection model says the duplicate-prediction failure mode
# survived GRPO in the released weights. It is suppressed at decode time.

results = rex.inference(images=image, task="detection", categories=[
    "man", "woman", "yellow flower", "sofa", "laptop", "cup", "lamp",
])

Caption: A detector that needs a repetition penalty is a detector that can still loop. GRPO cut duplicates from 1.23% to 0.08% of predictions, not to zero, and the default config covers the remainder.

Quantization is not the precision bottleneck

def to_bin(value_px, extent_px):
    """Pixel → one of 1,000 relative bins. This is the entire coordinate codec."""
    return min(999, int(value_px / extent_px * 1000))

def to_px(bin_idx, extent_px):
    return bin_idx / 1000 * extent_px

# Round-trip error at realistic resolutions:
for w in (640, 1024, 1920):
    print(w, "px wide →", w / 1000, "px per bin")
# 640  px wide → 0.64 px per bin   ← sub-pixel
# 1024 px wide → 1.02 px per bin
# 1920 px wide → 1.92 px per bin

# ← THIS is the trick, and the trap. The codec resolves BELOW one pixel on a
# 640px image, so discretization cannot explain the IoU=0.95 gap. The model is
# not being clipped by the grid. It is failing to select the right bin.
# That distinction is why a geometry-aware reward did not rescue it.

Caption: The 1,000 bin grid is finer than a pixel at COCO resolutions. Every explanation that blames quantization for the tight-IoU gap is arithmetically wrong.

The token budget, which is the real serving constraint

# Measured by the authors on 100 COCO images (paper section on inference efficiency)
REX_TOK_PER_BOX,  REX_TOK_PER_IMG  = 7.6,   45.3
SEED_TOK_PER
_BOX, SEED_TOK_PER_IMG = 148.8, 631.0

print(SEED_TOK_PER_IMG / REX_TOK_PER_IMG)   # 13.9x shorter output per image
print(SEED_TOK_PER_BOX / REX_TOK_PER_BOX)   # 19.6x cheaper per box

# Dense200 averages 91.2 boxes per image:
print(91.2 * REX_TOK_PER_BOX)    #    693 tokens  → fits max_tokens=2048
print(91.2 * SEED_TOK_PER_BOX)   # 13,571 tokens  → needs a 7x larger budget

# The default ceiling: 2048 / 7.6 ≈ 269 boxes per image before truncation.
# For an AUTOREGRESSIVE detector these are sequential decode steps, not
# parallel head outputs. Output length is latency, one to one.

Caption: The four-token box is not a storage optimization. In an autoregressive detector, every saved token is a saved sequential decode step, which is why the coordinate encoding is a serving decision rather than a formatting one.

It In Action

Input: the repo's cafe.jpg, queried with eleven categories in a single pass, including compositional phrases a category-level detector cannot express.

categories = ["man", "woman", "yellow flower", "sofa", "robot-shope light",
              "blanket", "microwave", "laptop", "cup", "white chair", "lamp"]
results = rex.inference(images=image, task="detection", categories=categories)

Step one, prompt assembly. All eleven phrases join into one natural-language query. Grounding DINO would need the categories fed as a concatenated text prompt and would return every chair regardless of the word white. Rex-Omni carries the modifier through the LLM.

Step two, vision encoding. Native-resolution ViT, patch size 28, image tokens bounded between 16 and 2560. No feature pyramid, which is exactly why small objects stay hard.

Step three, decode. One greedy stream:

<|object_ref_start|>man<|object_ref_end|><|box_start|><112><243><298><721><|box_end|>,
<|object_ref_start|>white chair<|object_ref_end|><|box_start|><604><388><772><690><|box_end|>,
<|object_ref_start|>microwave<|object_ref_end|><|box_start|>None<|box_end|>, ...

Absent categories return None rather than a hallucinated box, which is the behavior GRPO's F1-shaped reward pays for directly.

Step four, decode back to pixels. <112> on a 1024px-wide image is 114.7px. Sub-pixel codec resolution, as computed above.

Step five, the numbers. On COCO-scale scenes this costs 45.3 output tokens per image at 7.6 per box, roughly six boxes. The same workload through SEED1.5-VL costs 631.0 tokens. Fourteen times the sequential decode work for a lower F1 at every IoU threshold in the table.

Why This Design Works, And What It Trades Away

It works because IDEA already owned the hard part. The architecture is a vocabulary trick anyone could implement in an afternoon. The 22 million supervised samples, assembled by chaining Qwen2.5-VL-7B, DINO-X, Molmo, SAM and PaddleOCR into four automated annotation pipelines, are the actual product.

What it trades away, in order of how likely it is to block you:

No confidence scores. The paper switches from mAP to F1 because multimodal models lack reliable confidence estimation. That is not a metric footnote, it is a missing capability. Every regression detector in the comparison table was evaluated at a tuned score threshold, so those numbers are best-case per benchmark. Rex-Omni has one operating point and no dial. If your application needs high recall for triage or high precision for auto-labeling, a traditional detector lets you move; this does not.

Licensing. IDEA License 1.0 layered on the Qwen RESEARCH license, not Apache 2.0. The project markets itself as fully open-sourced. Check with counsel before shipping it commercially.

Latency scales with object count. A YOLO emits every box in one forward pass. Rex-Omni emits them one token at a time. Grounding DINO 1.5 Edge hits 75.2 FPS under TensorRT. A 3B autoregressive model on a 91-box scene is not in that conversation, and no amount of token efficiency closes a structural gap between parallel heads and sequential decode.

The supervision has a built-in ceiling. The grounding engine deliberately discards adjectives, because DINO-X mislabels them: prompt it with green lemon and it boxes every lemon. So three million grounding images are annotated with bare nouns by construction, stripping out precisely the compositional language the model is meant to master. Attribute grounding has to come from the referring engine and public referring sets instead. The student is bounded by its teachers, and the pipeline knows it.

No multi-scale features. The paper is candid that MLLMs lack feature pyramids, and dense tiny objects show it. F1@IoU 0.95 on Dense200 is 10.3.

Technical Moats

The moat is not the model. It is the annotation chain and the ten years of detection assets feeding it.

Reproducing Rex-Omni means reproducing 22 million labeled samples: three million grounding images through a caption, extract, filter, ground pipeline; three million referring images through a generate, point, mask, associate pipeline; five million geometric point conversions; two million OCR annotations. The grounding step requires DINO-X, which IDEA built and does not open source at the tier used here. The referring step requires Molmo and SAM. Then 12,288 A100 hours of SFT on top.

Second, lineage compounds. T-Rex, T-Rex2, ChatRex, RexSeek, Grounding DINO, Grounding DINO 1.5, DINO-X, and now Rex-Omni. Each generation's model becomes the next generation's annotator. They also own HumanRef, the referring benchmark Rex-Omni is evaluated on, which is a position worth noticing when reading the results.

Third, the zero-parameter design is strategically cheap to maintain. Because the coordinate tokens are repurposed rather than added, a future Qwen backbone can be re-derived without redesigning anything.

Insights

Insight One: the geometry-aware reward did not improve geometry.

The abstract states that GRPO with geometry-aware rewards bridges the discrete-to-continuous coordinate prediction gap and improves box accuracy. Put the SFT-only and GRPO checkpoints side by side at the threshold where box accuracy actually lives:

Benchmark

Rex-Omni-SFT

Rex-Omni (GRPO)

Delta

COCO F1@IoU 0.5

68.2

72.0

+3.8

COCO F1@IoU 0.95

15.8

15.9

+0.1

LVIS F1@IoU 0.5

60.3

64.3

+4.0

LVIS F1@IoU 0.95

20.7

20.7

0.0

Dense200 F1@IoU 0.95

10.6

10.3

−0.3

At IoU 0.5 the reward is worth roughly four points. At IoU 0.95 it is worth nothing, twice, and is negative once. An IoU reward that leaves tight-IoU performance unchanged is not tightening boxes. It is rejecting wrong ones.

The paper half-admits this: its own subsection on coordinate precision carries a question mark in the title. And the codec math rules out the convenient excuse, since 1,000 bins resolve to 0.64px on a 640px image. The gap is a representation-learning limit, not a discretization limit, and reward shaping at the sequence level does not reach it.

This also predicts where GRPO pays best. OCR on HierText goes 23.5 to 45.9, nearly doubling, because OCR scores demand exact text plus exact region and are annihilated by duplicates and misses. The more a benchmark punishes bad output structure, the more GRPO returns. The more it punishes bad geometry, the less.

Insight Two: no MLLM in the table survives contact with a strict threshold, and Rex-Omni is no exception.

Compute [email protected] as a fraction of [email protected] for every model the paper evaluates on COCO. The three strongest detectors retain 30.7%, 33.0% and 33.6%. Every multimodal model lands between 19.7% and 24.5%, and Rex-Omni sits at 22.1%, mid-pack among MLLMs and below every top-tier detector.

Being honest about the mess: some older detectors fall into the MLLM band too, DAB-DETR at 20.0% and Faster R-CNN at 11.7%. So this is not a clean architectural law. What is clean is the ceiling. No multimodal model tested exceeds 24.5% retention, while three detectors clear 30%. Rex-Omni's contribution raises the loose-threshold number and leaves the ratio where it was.

The practical rule: if your acceptance criterion is a human looking at a crop, this replaces your detector today. If it is a robot gripper closing on a box edge, or a measurement pipeline, it does not, and the paper's own numbers say so.

Takeaway

On VisDrone, deduplicating the SFT model's output scores 62.3. The GRPO model scores 61.6. A post-processing filter you could write in twenty lines beat 192 GPU hours of reinforcement learning.

The paper reports that removing repeated predictions lifts the SFT model from 55.6 to 62.3, discarding 15.3% of its outputs, while the GRPO model moves 61.6 to 62.1 with 0.1% removed. Read straight across: SFT plus a dedup pass edges out fully GRPO-trained inference on that benchmark.

The nuance that saves the method: this only works where the failure is redundancy. On Dense200 the failure mode is large-box collapse, a single box swallowing the scene, and dropping those predictions leaves the SFT model at 56.7 against GRPO's 60.0. You can filter a duplicate because the correct answer is still in the output. You cannot filter an absence.

So the honest scope of what GRPO bought is narrower than the abstract implies: it removes a failure mode you could already post-process, and it removes one you could not. Before spending on an RL stage, check which of the two you actually have.

TL;DR For Engineers

  • Rex-Omni adds zero parameters to Qwen2.5-VL-3B. It repurposes the final 1,000 vocabulary tokens as coordinate bins, making a box four tokens instead of fifteen.

  • GRPO cost 192 A100 hours against SFT's 12,288, which is 1.6% of the compute and 0.3% of the data, and produced the headline gain by fixing output behavior.

  • The geometry reward did not improve geometry: COCO F1@IoU 0.95 moved 15.8 to 15.9, LVIS did not move, Dense200 went down.

  • Measured 7.6 output tokens per box against SEED1.5-VL's 148.8, a 13.9x shorter response per COCO image. In an autoregressive detector that is latency, not just bytes.

  • No confidence scores means no tunable precision and recall operating point, and the license is IDEA 1.0 over Qwen Research, not Apache. Both are shipping blockers before any benchmark is.

Explain It Like I'm New

For a decade, teaching a computer to find objects in a picture meant training a specialist. You gave it a fixed list of things to look for, and it learned to draw tight rectangles around them. They became excellent at drawing and poor at understanding. Ask one for a red apple and it will happily box every apple in the bowl, because it never really learned what red means. It learned what apples look like.

Language models have the opposite problem. They understand phrases like the third chair from the left, but historically could not point at anything accurately.

Rex-Omni closes the gap with an idea that sounds almost too simple. A language model already predicts the next word from a list of known words. So borrow a thousand unused entries from its vocabulary and declare that they now mean positions, from the far left edge to the far right. Detection becomes writing a sentence where some of the words happen to be places.

This matters because it collapses many separate systems into one. Reading text in a photo, finding a button in a screenshot, marking joints on an animal, pointing at empty space on a shelf: previously four models, now four phrasings of the same request.

The catch is precision. This approach finds the right object reliably and draws a slightly looser box around it than a specialist would. Whether that is acceptable depends entirely on whether a person or a machine consumes the result.

See It In Action

  • Hugging Face Space, live demo (Mountchicken/Rex-Omni), the fastest honest test. Feed it a compositional phrase like white chair or the person on the left and watch whether the modifier survives. That single test separates this from open-vocabulary detectors better than any benchmark table.

  • Task cookbooks and notebooks (tutorials directory), standalone scripts plus Jupyter notebooks for all eight task modes. The GUI grounding example raises max_tokens to 4096 while detection uses 2048, which quietly documents where the output-budget pressure sits.

  • Fine-tuning guide (finetuning/README.md), both SFT and GRPO stages, TSV-backed datasets with base64 images and a line-index file. Pointing-task fine-tuning was added in January. This is the most useful artifact in the repo if you intend to adapt rather than consume.

  • Grounding DINO and T-Rex2 repos (Grounding DINO, T-Rex2), the systems Rex-Omni is measured against, both from the same lab. T-Rex2 still beats it on visual prompting and counting, and understanding why sharpens the whole comparison.

  • PaddleOCR (repo), simultaneously the OCR baseline in the paper and the annotator that produced Rex-Omni's two million OCR training samples. Worth sitting with when reading the OCR table.

Community Conversation

  • CVPR 2026 acceptance, now carried in the repo title, the strongest available third-party signal. Peer review cleared the claims, which is more than most benchmark-heavy releases get before the community weighs in.

  • IDEA's own framing on the RexSeek repo asks whether you are still using traditional detectors and calls this the next generation of perception models. Notable because RexSeek's own pipeline needs Grounding DINO to propose boxes first. The lab is publicly deprecating its own architecture, which is a real signal about internal conviction.

  • Hugging Face model discussions, early activity is largely packaging and metadata rather than benchmark disputes, including a staff PR adding library_name and license fields. Adoption friction, not accuracy friction.

  • AWQ quantization shipped two weeks after release, halving storage, and pointing fine-tuning landed in January. The maintenance cadence suggests real deployment interest rather than a paper drop.

  • Independent write-ups such as this technical walkthrough repeat the 72.0 F1 headline while noting the IoU 0.95 nuance. The loose-threshold number is what travels; the retention ratio is not being discussed anywhere, which is the gap this issue exists to fill.

Detection Stopped Being A Geometry Problem And Became A Vocabulary One

The useful thing Rex-Omni proves is not that a 3B model can out-detect DINO. It cannot, at any threshold that demands tight boxes. What it proves is that the localization problem was never the interesting half. Language-aware perception was, and the price of admission turned out to be a thousand recycled token IDs and a very expensive annotation pipeline.

The uncomfortable part is what the ablations say about method attribution. A reinforcement stage described in terms of geometry-aware rewards produced its gains by regulating output structure, and on one benchmark a deduplication filter matched it outright. That is not a criticism of the result, which is real and reproducible from the paper's own tables. It is a warning about reading mechanism claims from headline deltas.

Check which failure mode you have before you buy the machinery to fix it.

References

Rex-Omni reformulates detection, OCR, pointing, GUI grounding and keypointing as one next-token problem by repurposing the last 1,000 vocabulary entries of Qwen2.5-VL-3B as quantized coordinates, adding no parameters and cutting a bounding box to four tokens. Its GRPO stage, at 1.6% of the SFT compute, delivered the headline gains by correcting duplicate and large-box behavior rather than by improving geometric precision, which stayed flat at strict IoU thresholds. It matters because it moves the frontier of visual perception from box regression toward language-grounded querying, while leaving a measurable precision ceiling that decides whether you can deploy it.

Keep Going

The transferable lesson is an attribution habit. When a paper credits a mechanism, find the metric that mechanism should move and check it directly. Here the reward was geometric and the geometry metric did not budge, which reframes the entire contribution in about two minutes of arithmetic.

SnackOnAI runs this teardown weekly on the systems engineers actually deploy, model architectures, serving stacks, kernels, and the benchmark columns vendors leave out of the abstract. No announcements, no press release summaries. Subscribe at snackonai.com and join 10,000+ engineers reading it.

Forward this to whoever on your team is about to replace a detector with an MLLM.

Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad, it helps us keep building and delivering value 🚀

Smarter CRM. Less Busywork.

Disconnected data and tools make it harder to understand your customers. HubSpot's Agentic Customer Platform brings your data, teams, and tech stack together with AI built in to help your business work faster and create more personalized customer experiences.

Why HubSpot and what's new

  • Use AI powered tools to take action faster

  • Unify your data, teams, and tech stack in one place

  • Create one shared view of customer data

  • Connect teams around the same customer context

  • Bring your business tools into one place

Connect more of your business in one place and give every team a smarter way to work. Get set up quickly and start checking off your hardest tasks.

Get started free

Recommended for you

View all
caret-right
© 2026 Snack On AI.
beehiivPowered by beehiiv