HySparse2: How Xiaomi Split the Transformer to Fix Agent Prefill Costs

On September 22, 2026, researchers from Xiaomi’s LLM-Core team (Jianyu Wei, Yizhao Gao, Qihao Zhang, Shimao Chen, Zhengju Tang, and colleagues) posted HySparse2 (arXiv:2609.26368) to arXiv. The paper introduces a hybrid sparse attention architecture specifically designed around a property of long-running agents that existing Transformers handle badly: agents read enormous amounts of tool output and environment observations, then emit very short actions. HySparse2 cuts prefill computation by 5.02 times and KV-cache storage from 12.09 GB to 2.69 GB at one million tokens on a 80B-A3B MoE model, without giving up meaningful accuracy on agentic retrieval benchmarks.

Why Agentic Workloads Break Standard Transformer Economics

Every round of a multi-turn agent interaction follows the same asymmetric pattern. The agent generates a short tool call, maybe twenty tokens. The tool returns a search result, execution trace, or document, maybe ten thousand tokens. Before the agent can respond, those tokens need to run through all the model layers during prefill. As the conversation accumulates, each new round of prefill must process an expanding context that grows linearly with interaction depth.

The problem is that standard Transformer architectures handle “1 million tokens read” and “ten tokens generated” with essentially the same stack of identical decoder blocks. Every layer computes attention over the full context, maintains its own KV cache, and contributes to the overall latency and memory budget. That is fine when inputs and outputs are similar in length. It becomes increasingly wasteful when inputs are orders of magnitude longer than outputs.

HySparse addresses part of this by interleaving full-attention layers with sparse-attention layers within a standard decoder. Each full-attention layer provides selection indices and a shared KV cache to the sparse layers that follow it, reducing how much attention computation the sparse layers need to do. The prefill path still runs every layer of the model, however. HySparse2 goes further by splitting the model into two halves with different roles and allowing prefill to exit after the first half.

The Architecture: Two Decoders, Two Levels of KV Sharing

HySparse2 borrows its overall structure from YOCO (Sun et al., 2024), which divided the decoder into a self-decoder and a cross-decoder. HySparse2 applies this idea to a hybrid attention model and adds a second level of KV sharing on top.

The Self-Decoder and Cross-Decoder Split

The backbone has 49 Transformer layers. The first 25 form the self-decoder, which uses hybrid attention combining full attention and sliding-window attention (SWA) for local context modeling. The remaining 24 layers form the cross-decoder, which combines full attention and sparse attention for global retrieval. The interesting part is what happens at the boundary between them.

During prefill, the model runs only the self-decoder. The cross-decoder does not execute. Instead, the KV caches that the cross-decoder would need during decoding are constructed from the self-decoder’s hidden states using learned projections. The paper describes this as “KV Bridging.” Each full-attention layer in the cross-decoder has its own K/V projections that take the corresponding self-decoder full-attention layer’s hidden states as input:

K_j^cross = Proj_j^K(H_i^self)
V_j^cross = Proj_j^V(H_i^self)
Q_j^cross = Proj_j^Q(H_j^cross)

The query projection still uses the cross-decoder’s own current hidden states. The effect is that each cross-decoder full-attention layer gets distinct KV representations derived from the self-decoder, even when multiple cross-decoder layers share a single self-decoder hidden state source. Prefill therefore runs only the first 25 layers of the 49-layer model before producing all the KV state needed for decoding, a path that “can exit after the self-decoder, skipping all cross-decoder layers.”

The important design constraint is that KV Bridging operates only on full-attention layers, not SWA layers. Earlier work (YOCO) used cross-layer sharing for all layers, but a separate SWA branch in the cross-decoder creates a dependency on the cross-decoder’s own hidden states that grows with depth and cannot be skipped. HySparse2 eliminates the SWA branch from the cross-decoder entirely and instead forces recent tokens into the sparse selection budget (described below), removing that cascading dependency by construction.

KV Reuse and Token-Level Sparse Selection

Within each hybrid block in the cross-decoder, HySparse2 keeps HySparse’s original KV Reuse design: the full-attention layer computes attention over the whole context and produces both a KV cache and token selection indices; the sparse-attention layers that follow reuse those indices without recomputing them. The refinement in HySparse2 is changing the selection granularity from blocks of 64 tokens to individual tokens.

The motivation is specific to agent trajectories. Agent conversations interleave reasoning segments, tool calls, and returned observations. The tokens that contain relevant evidence for a given response can be scattered across the context, with role delimiters and special tokens between them. Block-level selection allocates the sparse attention budget to a block of 64 tokens at once; if one relevant token sits next to 63 irrelevant neighbors, all 64 are retrieved. Token-level selection allocates the budget to 1,024 individual tokens across the full context, plus a forced window of the 128 most recent tokens. The ablation confirms this: token-level selection improves RULER-v2 by 6.57 points, two-needle MRCR-v2 by 8.14 points, and GraphWalks by 5.55 points under the same attention budget at the 32k context length used during pretraining.

The forced local window replaces the separate SWA branch that HySparse used in its cross-decoder sparse layers. Rather than maintaining a second attention branch with its own projection weights and KV cache, HySparse2 simply forces the 128 most recent tokens to always appear in the sparse selection, alongside the 1,024 globally selected tokens. Those tokens are read from the full-attention KV cache already stored by the preceding full-attention layer. No additional KV state is needed in the cross-decoder. This is what makes the complete cross-decoder early exit feasible.

Results: What the Numbers Actually Say

The experiments compare HySparse2 against HySparse and Hybrid SWA on identical 80B-A3B MoE models trained on the same data and schedules. The models differ only in their attention design. Hybrid SWA uses 9 full-attention layers; HySparse and HySparse2 each use 5.

Prefill cost at 1 million tokens (FP8 KV cache):

HySparse2: 2.69 GB KV cache, 2.92× lower prefill FLOPs than HySparse, 5.02× lower than Hybrid SWA
HySparse: 6.72 GB KV cache
Hybrid SWA: 12.09 GB KV cache

Pretraining long-context performance (80B-A3B models):

RULER: HySparse2 90.77 vs HySparse 84.89 vs Hybrid SWA 88.71 (HySparse2 +5.88 over HySparse)
NoLiMa: HySparse2 49.76 vs HySparse 40.27 vs Hybrid SWA 30.13 (HySparse2 +9.49 over HySparse)
Repo Code PPL: HySparse2 1.1570 vs HySparse 1.1588 vs Hybrid SWA 1.1578 (lower is better)

Post-training retrieval and agent perplexity (after ~100B tokens of light post-training at 256k context):

Mean MRCR-v2 (multi-round retrieval): HySparse2 +11.30 pp over HySparse, +6.44 pp over Hybrid SWA
Mean RULER-v2 (12 retrieval subtasks): HySparse2 +19.81 pp over HySparse, +18.65 pp over Hybrid SWA
At 256k context, RULER-v2: HySparse2 58.45 vs HySparse 32.61 vs Hybrid SWA 35.74
AgentPPL and LongPPL: HySparse2 is lower (better) than both baselines at all evaluated lengths up to 256k

On general capabilities (MMLU, BBH, MATH, GSM8K, code benchmarks), the picture is mixed, as expected. HySparse2 improves on BBH (64.29 vs 61.93 for HySparse) and MMLU-Pro (37.56 vs 35.74), while HySparse retains advantages on DROP (63.78 vs 58.99) and GSM8K (64.14 vs 61.94). The authors present this as “broadly comparable,” which is accurate: the improvements are concentrated where the architecture was designed to improve.

KV Bridging Ablation at Scale

One concern with KV Bridging is whether the cross-decoder’s reliance on self-decoder hidden states degrades model quality relative to a model that constructs its own KV caches normally. The ablation addresses this at the 290B-A8B scale, training models with and without KV Bridging on approximately 1.8 trillion tokens. The result: “MMLU and TriviaQA improve slightly, RULER changes by only 0.31 points, and Repo Code PPL is nearly unchanged. LongPPL improves from 3.6053 to 3.4202. BBH and GSM8K decrease by about one point, and DROP falls from 71.37 to 68.17.” The authors consider this an acceptable trade-off given the prefill savings. Whether the DROP regression is acceptable depends on the intended use case.

Limitations and Open Questions

This is a research preprint. No production model checkpoint or inference runtime tied to HySparse2 has been released. All benchmark figures come from the authors’ own experiments; independent replication has not happened yet.

The efficiency gains at 1 million tokens are analytically sound (shorter prefill path means fewer FLOPs), but how they translate through production inference engines and real GPU hardware depends on kernel implementations that are not described in the paper. Token-level sparse attention requires custom kernels to be efficient; the paper cites Wang et al. (2025) for recent advances in sparse kernels that make token-level selection practical, but the performance figures are theoretical FLOPs counts, not wall-clock latency measurements on specific hardware.

The DROP regression is the clearest signal that KV Bridging introduces a quality trade-off on tasks that require multi-step numerical reasoning over shorter contexts. If an application is primarily DROP-style reasoning at modest context lengths, the benefits of the architecture do not clearly justify the accuracy cost. The ablation on the Forced SWA design (replacing the separate SWA branch with a forced local window) also shows a 5.08-point GSM8K regression and a 4.99-point MRCR-v2 regression compared with keeping the Gated SWA design, though at the cost of making early exit impossible. HySparse2 makes a deliberate choice in favor of prefill efficiency over those benchmarks.

What This Means for Engineering Teams

The immediate serving implication is in prefill-decode disaggregation. Under this serving pattern, the HySparse2 prefill node hosts only the self-decoder and KV Bridging projections, specifically the first 25 of 49 layers. The paper states this “cuts the memory requirement of the prefill node by nearly half.” For teams running long-context inference at scale, where prefill nodes are often GPU-memory-bound, that is a concrete hardware cost reduction.

The deeper architectural implication is about how to think about model design for agents. Current Transformer designs treat a 1-million-token observation and a 10-token action as inputs to the same homogeneous stack of decoder blocks. HySparse2 starts to separate reading architecture from action architecture. The self-decoder can be optimized for efficiently processing long observations; the cross-decoder handles the reasoning and generation steps that need access to that processed context. This is closer to how encoder-decoder models worked before decoder-only became dominant, but applied asymmetrically based on actual agent workload shape rather than input/output type.

For teams interested in long-context attention research or evaluating model architectures for production LLM deployments, HySparse2 is worth watching carefully. The token-level sparse selection improvement is usable independently of the KV Bridging change, and the ablation results suggest it provides consistent retrieval gains. The KV Bridging and early-exit design require training from scratch or substantial architectural changes, which limits how quickly engineering teams can adopt them. But they describe a direction that production inference providers will likely implement once the kernel and tooling support matures. For a detailed look at KV cache compression techniques and where they fit in the inference stack, the tradeoffs HySparse2 navigates are directly relevant.

Key Takeaways

HySparse2 (arXiv:2609.26368, Jianyu Wei, Yizhao Gao et al., Xiaomi LLM-Core, September 22, 2026) splits the Transformer into a self-decoder and cross-decoder, allowing prefill to run only the first half of the model.
At 1 million tokens with FP8 KV cache, prefill FLOPs drop 2.92× vs HySparse and 5.02× vs Hybrid SWA; KV-cache storage drops from 12.09 GB to 2.69 GB.
After light post-training, RULER-v2 improves by 19.81 percentage points over HySparse and 18.65 over Hybrid SWA; at 256k context, the gap is 58.45 vs 32.61 vs 35.74.
Token-level sparse selection (replacing block-level) improves RULER-v2 by 6.57 points and two-needle MRCR-v2 by 8.14 points under the same attention budget.
Under prefill-decode disaggregation, the prefill node hosts only the self-decoder, cutting prefill node memory requirements by nearly half.
The architecture introduces regressions on DROP (-3.2 pp) and GSM8K (-1 pp); this is a research preprint with no production checkpoint released yet.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If your team is designing long-context agent architectures and needs guidance on model selection and inference stack trade-offs, talk to our team.