Colibrì Brio: Score Closed Options Instead of Generating Tokens

Colibrì v1.12.0, released September 20, 2026, ships a new inference primitive called Brio mode. Instead of asking the model to write an answer and then parsing what it wrote, the caller supplies the allowed options and the runtime scores their probabilities. The response has completion_tokens=0. The model cannot return an option outside the supplied set. No generation, no parsing, no retry loop for malformed output.

The interesting part is that Brio does not require a specialist classifier. The same foundation-model checkpoint that handles open-ended generation now exposes a second computational mode at the serving layer. What changes is the API contract, not the model weights.

The Problem Brio Solves

A large fraction of the calls that land on LLM endpoints in production agent systems are, structurally, if-statements. The agent needs to know whether a document is relevant, which of four tools to call next, what sentiment category applies, or whether a value falls into one of three risk buckets. The model gets asked to generate the answer, and the generation is a token string that gets parsed against an expected vocabulary.

This is expensive in several ways. The model generates tokens one at a time. Parsing can fail, requiring a retry. Constrained decoding grammars add overhead and can affect what the model actually prefers to say. And when you need to ask ten classification questions against the same document, you re-process the entire document context ten times.

The conventional alternative, training a specialist classification model for each decision type, trades inference cost for training cost and creates a maintenance problem: every new decision type needs a new model. The TypeSafe Jev approach addressed structured output at the type level. Brio addresses it at the inference primitive level, using the same generative checkpoint you already run.

How Brio Mode Works

Brio is opt-in per request through two submit keys: logprobs=k and pin=1. Without these keys, every request is byte-identical to the previous behavior, which the Colibrì release notes say is asserted per engine rather than claimed. Setting max_tokens=0 is now legal, but only together with logprobs>0.

The shared contract across all nine Colibrì engines lives in three headers: decode_batch.h for the logprob tail, serve_codec.h for the wire format, and pin_pool.h for nested KV-state snapshots. The snapshot pool is the mechanism that makes multi-question Brio efficient.

KV Snapshots: Shared Context Without Re-Reading

When you send multiple closed questions against the same document, the model only has to read that document once. The snapshot pool takes a photograph of the KV state at a specific prefix depth. For a single-level snapshot, each option causes the question to be re-read once before the option tokens are evaluated. For a two-level snapshot, the instructions are photographed once and the question plus instructions are photographed per question, so option evaluation draws from the deeper snapshot without repeating either layer.

The Colibrì release notes give a concrete count: four items with two-level snapshots cost 176 processed tokens instead of 496. That is a 2.8x reduction in context overhead, not in the model’s underlying compute per forward pass, but in how many tokens are fed into the KV cache across all four evaluations.

The guard that makes this safe is kv_prefix_holds(). A snapshot is reused only when the prefix record still holds the same IDs, because a KV bank can be dropped between two option evaluations. If the record no longer matches, the system falls back to cold recomputation rather than reading from stale state. The release notes state that logprobs computed from a snapshot match cold recomputation to 0.00e+00 on all nine tiny test fixtures and on two real checkpoints: qwen36 at 22 GB and DeepSeek V4.1 Flash at 476 GB.

The Three Request Forms

Brio exposes three endpoint shapes, each suited to a different pattern in agent and pipeline code.

options is one closed question. Supply a list of allowed values; receive a probability for each. This is the simplest form and the most direct replacement for a generate-then-parse call that picks between a small set of outcomes.

questions is many questions on one state. Each question carries its own option set. The server photographs the shared context state once and orders the snapshot evaluations internally. The claimed speedup for this form is 5.7x. The practical pattern here is classification-heavy pipelines: route a document through ten binary or multi-class decisions in one request, drawing from one shared KV state.

schema is the most structurally different form. The caller supplies an object of field names to allowed values. The server writes the JSON skeleton and fills each field one at a time from model probabilities. The resulting object is valid by construction because the server assembled it, not the model. Each field value also carries an entropy measurement alongside its probability. The claimed speedup for the schema form is 2.4x, which is lower than questions because schema field evaluation requires sequential KV state construction rather than fully parallel evaluation.

Accessing Brio

Three clients reach the endpoint: POST /v1/brio on the gateway, /brio in coli chat, and a dedicated dashboard page that reads a document once and then asks it several questions, each with its own option set. The complete request format, reply structure, normalization choices, and cases where Brio does not help are documented in docs/brio.md in the repository.

Brio runs across all nine Colibrì inference engines, as detailed in the v1.12.0 release notes. The v1.12.0 release also ships a performance improvement to the MoE kernel in expert_ffn.h: int4 weights stay planar in RAM instead of being unpacked to int8, and two OpenMP regions per layer replace three per expert. This takes qwen36 from 12.8 to 15.7 tok/s with the resident set falling from 29 to 17 GB.

Limitations and Open Questions

Brio reduces context re-processing and eliminates answer decoding. It does not reduce the FLOPs the model spends on the actual forward pass for each option evaluation. The raw computational cost of running a 22 GB or 476 GB model remains, which means Brio changes the software primitive more clearly than it changes hardware requirements.

The speedup figures, 5.7x and 2.4x, are Colibrì’s own measurements against its own test fixtures. The release notes are precise about the conditions: the token counts are documented for the specific case of four items, and the project is careful to note that the cases where Brio does not help are in its own documentation. Whether the speedup profile holds for larger option sets, longer documents, or different model families is not established in the release notes.

Brio also does not eliminate the fundamental limitation that the model only evaluates the options you supply. If the correct answer is not in the option set, the model will score confidently against a wrong option. This is not a defect in Brio specifically, but it means that the quality of the option set design matters as much as the inference mechanism.

Academic work on structured LLM output suggests that syntactic validity does not guarantee semantic accuracy. A cited comparison from the JSONSchemaBench study found 91.37% accuracy with constrained decoding versus 93.63% for unconstrained generation with post-hoc parsing on function-calling tasks. Brio is not constrained decoding in the grammar-guided sense, but the general warning applies: scoring probabilities against a closed set does not substitute for evaluation against ground truth.

What This Means for Engineering Teams

Brio’s practical value sits at a specific point in the agent stack: any call that is structurally a multi-class decision. Route selection, topic classification, relevance scoring, policy choices, sentiment binning, form field extraction from constrained vocabularies. These are currently implemented as generate-then-parse calls with retry logic, often with constrained decoding overhead added on top.

Replacing those calls with Brio requests against the same checkpoint that handles generation removes a category of parsing bugs and retry logic from the application code. The schema form handles the structured extraction case that would otherwise require either a JSON-mode generation call or a specialist extraction model.

For teams running LLM infrastructure, the KV snapshot pooling design also has broader implications. The same foundation-model checkpoint can now serve open-ended generation requests and bounded probabilistic scoring requests from a shared KV cache. That means a single model deployment can handle both modes, which changes the capacity planning question from “how many specialist models do I need” to “how do I partition my shared foundation model’s capacity across request types.”

Teams evaluating LLM inference architectures should consider whether any existing classify-or-route calls in their pipelines are candidates for this pattern. The migration path is low risk: Brio is opt-in per request, the behavior without the keys is byte-identical to current behavior, and the correctness claim is asserted at the engine level against two real production-scale checkpoints.

Key Takeaways

• Brio mode in Colibrì v1.12.0 evaluates option tokens rather than sampling them, returning probabilities with completion_tokens=0 and no answer outside the supplied set.
• Nested KV snapshots reduce four-question context overhead from 496 to 176 processed tokens, a 2.8x reduction.
• The three request forms are: options for one closed question, questions for many questions on one shared state (5.7x speedup), and schema for structured field extraction with per-field probability and entropy (2.4x speedup).
• Correctness is measured against cold recomputation to 0.00e+00 on qwen36 (22 GB) and DeepSeek V4.1 Flash (476 GB).
• The same foundation-model checkpoint serves both generative and Brio requests; no separate classifier model is required.
• Brio is opt-in per request and byte-identical to current behavior without the keys, making adoption incremental.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If your team runs agent pipelines where classification, routing, or structured extraction calls are generating more tokens than answers, talk to our team.