SPLASH: Change LLM Serving Topology Mid-Request Without Draining

Researchers at the Institute of Computing Technology, Chinese Academy of Sciences, have built a serving system that switches how attention is parallelized mid-request, without stopping or draining the running batch. The system is called SPLASH (Switching Parallel Layouts of Attention with Seamless Handoff), published September 29, 2026 by Chuan Liu et al. (arXiv:2609.37626). It matters now because reasoning, agentic, and reinforcement-learning rollout workloads routinely change shape while running: a batch that begins as hundreds of short requests ends as a handful of very long traces, and no fixed attention layout handles both extremes well.

Why Fixed Parallelism Is a Compromise

Every large-scale LLM deployment makes one decision at launch: which attention parallelism strategy to run. The choice is real because the right answer genuinely depends on traffic shape, and traffic shape changes.

Tensor parallelism (TP) shards each attention projection weight across ranks. For a small number of concurrent requests, that weight distribution keeps individual matrix multiplies large and efficient. But TP replicates the KV cache on every rank, meaning each GPU holds a full copy. That replication does not hurt when contexts are short; it becomes expensive as contexts grow, because every rank must store the entire KV history for every request it participates in.

Data-parallel attention (DP-attention) takes the opposite approach. Projection weights are replicated on every rank, and the KV cache is divided by request, each rank owning its subset. With many independent requests, the division distributes memory load well. But weight replication means that as the number of active requests drops, memory spent on redundant weight copies becomes a fixed tax that does not reduce.

Context parallelism (CP) divides the KV cache by token position rather than by request. It is the natural fit for a small number of very long sequences, distributing the growing positional history across ranks. For wide, short batches, CP also pays the weight-replication tax without getting the memory benefit it is designed for.

For a workload that starts wide and short and ends narrow and long, all three layouts are correct at different moments during the same set of requests. Serving engines have historically fixed the choice at launch because changing it required draining the batch and restarting workers. Understanding these foundational trade-offs is part of solid LLM inference fundamentals.

The RL rollout scenario makes this especially visible. Each rollout step launches thousands of simultaneous generation requests. Most finish quickly. A small number of long reasoning traces hold the entire group until they complete. A fixed deployment must provision for a compromise: enough capacity for the wide launch, efficient enough for the long tail. Neither regime gets its optimal layout.

The Core Observation: KV Cache and Weights Are Now Decoupled

The key insight in SPLASH is architectural rather than algorithmic. In traditional multi-head attention, the number of KV heads equals the number of query heads. Tensor parallelism shards along heads, and the KV cache is naturally indexed by the same head dimension. The cache and the weight sharding are coupled: you cannot move one without moving the other.

Modern attention architectures break that coupling. Grouped-query attention (GQA) uses fewer KV heads than query heads. Multi-query attention (MQA) uses a single shared KV head for all queries. DeepSeek-style multi-latent attention (MLA) compresses each token’s KV into one latent vector that all heads share. When KV heads number in the single digits, or reduce to one, sharding the projection weights along the query-head dimension no longer forces a corresponding split of the KV cache. The cache and the weight layout can vary independently.

This is the observation that makes SPLASH possible. If KV is decoupled from weight layout, you can describe any attention parallelism strategy as a combination of two choices: how projection weights are distributed across ranks, and how the KV cache is distributed. A layout switch becomes a matter of moving state to match the destination’s distribution, not a full restart.

Four Layouts and a New One: DOP

SPLASH formalizes four attention layouts. Three exist in prior systems. The fourth is new.

The per-rank memory footprints make the trade-offs precise. Let W_A be the size of attention projection weights, k be the KV bytes per token, B be the number of balanced requests, s be the context length, and T be the number of ranks:

For TP: M_TP(s) = W_A/T + B * k * s. Weights are sharded, so weight memory scales down with T. But the KV cache is replicated on every rank, so KV memory does not benefit from T at all. As s grows, the replicated KV dominates.

For DP-attention: M_DP(s) = W_A + B * k * s / T. Weights are replicated, so W_A is paid in full on every rank regardless of T. The KV is divided by request, so KV memory scales down. At high concurrency with short contexts, the weight replication is an acceptable trade. At low concurrency, it is pure overhead.

For CP: M_CP(s) = W_A + B * k * s / T. The formula is identical to DP-attention. CP differs from DP-attention in how the KV is divided (by position rather than by request), but both replicate weights and share the same memory structure.

The fourth layout is new: Decoupled Ownership Parallelism (DOP). DOP shards the projection weights across ranks AND keeps each request’s entire KV history on a single owner rank. The per-rank formula is: M_DOP(s) = W_A/T + B * k * s / T. Both terms scale down with T. There is no redundancy in either weights or KV. DOP provides the best memory scaling of any of the four layouts.

The interesting part is why DP-attention cannot match this. In DP-attention, each rank replicates the full projection weight W_A. For GLM-5.3 in FP8 at T=8 ranks, those replicated weights (Q-B, KV-B, and O projections across 78 transformer layers) consume 12.69 GiB per GPU that could otherwise hold KV cache. DOP frees exactly that 12.69 GiB per rank because it shards the weights instead of replicating them. That is not a free lunch: DOP pays a per-layer all-to-all communication to collect the right query rows before attention and another all-to-all after. But the volume of that exchange tracks the current batch’s query rows, not context length. As contexts grow, DOP’s communication overhead stays flat while DP-attention’s weight-replication tax stays constant.

The all-to-all exchange in DOP works as follows. Each rank holds attention projections for H/T heads, where H is the total head count. Before attention computation, each rank has N query rows for its H/T heads. To compute attention at an owner rank, that rank needs all N query rows for all H heads. A variable-size all-to-all collects the missing rows, attention runs on the full set, and a reverse all-to-all redistributes the outputs. The communication volume per layer is V_DOP = (1 – 1/T) * N * H * (d_q + d_z) * b bytes, where d_q is the query dimension, d_z is the latent dimension, and b is the byte size. That quantity grows with current batch size N, not with historical context s, which is the key property that makes DOP increasingly attractive as long requests displace short ones late in a rollout.

How Switches Happen Without Draining

A layout switch amounts to reconciling what each rank currently holds with what the destination layout requires. SPLASH reduces all directed switches between the four layouts (twelve total, since any layout can transition to any other) to three primitives.

The first is discard. When the destination layout requires less state than the current one, the surplus is simply freed. No network traffic required. Switching from TP to any layout that does not replicate KV falls partly into this category once the redundant copies are released.

The second is all-gather. When a rank needs blocks currently held by other ranks (for example, collecting the full KV history for a request that was previously partitioned by position under CP), SPLASH issues an all-gather to assemble the missing pieces.

The third is all-to-all. When ownership itself must change, such as when transitioning between request-based division (DP-attention) and position-based division (CP), an all-to-all exchanges KV blocks so each rank ends up with the right subset for the destination layout.

SPLASH moves state one transformer layer at a time, pipelined inside an ordinary serving step. While layer i-1 computes, layer i fetches the weights and KV it needs for the destination layout. The batch does not pause. Measured communication times for a switch on B200 GPUs with B=16 requests and 256K total tokens: 0.29 to 0.30 ms per layer for projection weights, 3.47 to 3.81 ms per layer for KV. End-to-end switch time from start to completion across the full model ranges from 0.02 ms to 11.76 ms depending on the transition direction and how much state must move. Without SPLASH, a blocking switch (drain the batch, reconfigure, restart) costs up to 667.31 ms. Median overhead with SPLASH is under 0.51% of the serving step during which the switch runs.

Performance Results

The scheduler that drives SPLASH monitors which layout leads the current batch and switches when another layout holds a sufficient throughput margin. In an RL rollout scenario, the scheduler runs DP-attention while the batch is wide (many short concurrent requests) and switches to CP once only the long-context tail remains.

On B200 GPUs serving GLM-5.3, end-to-end throughput improves 1.3 to 1.73 times over fixed-layout baselines. The same layout transition pattern appears in experiments with DeepSeek-V3.2 on H200 GPUs and with GLM-5.3-Flash on DCU accelerators, suggesting the result generalizes beyond any single GPU platform or model family.

The 1.3 to 1.73 times range reflects workload shape. Workloads with the most pronounced phase change see the larger end of the range because the fixed-layout baseline is more severely compromised across both extremes. DOP extends how long SPLASH can stay in a throughput-optimal regime for long-context requests: the freed 12.69 GiB per GPU makes room for more KV cache before eviction pressure forces a topology change.

Limitations and Open Questions

SPLASH’s core insight depends on the KV-weight decoupling that GQA, MQA, and MLA provide. What’s easy to miss here is the constraint this puts on applicability. Classic multi-head attention, where KV heads and query heads are equal, does not offer that decoupling. The authors note this explicitly for MLA on DeepSeek-style models: the latent is shared by all heads, so both TP and MLA-under-TP hold a full copy of the latent on every rank, and switching between them frees no cache. SPLASH works best on models where the KV representation is genuinely compact relative to the query representation.

The scheduler that determines when to switch uses a margin threshold. Choosing that threshold well requires workload knowledge. Too aggressive and SPLASH incurs switch overhead on transient fluctuations that would have resolved themselves. Too conservative and the system stays in a suboptimal layout longer than necessary. The paper does not provide a principled method for setting the threshold, and production workloads have more variance than the rollout scenario used in experiments.

The switch overhead measurements are reported at B=16 requests and 256K total tokens. How communication cost scales as context grows to millions of tokens (increasingly common for long-document and long-horizon reasoning workloads) is not characterized in the paper. KV volumes that are 10x or 100x larger than the measured scenario will push the all-gather and all-to-all times proportionally, and the sub-0.51% overhead figure may not hold across that range.

DOP’s all-to-all communication cost tracks N (current batch size), not context length. At very high batch sizes that volume grows proportionally, and the paper does not provide a threshold at which DOP’s communication overhead outweighs its memory advantage.

What This Means for Engineering Teams

The practical impact lands differently depending on what type of workloads a team is running. For teams serving standard inference with stable concurrency and predictable context lengths, a fixed layout chosen at deployment works well enough and SPLASH adds complexity without a proportionate return.

For teams running RL rollout pipelines, agentic pipelines with variable branching depth, or any workload where a single request batch goes through distinct phases (many short requests yielding to a few long ones), SPLASH is directly applicable. The 1.3 to 1.73 times throughput improvement reported on B200 GPUs translates into real capacity, which means fewer GPUs needed to hit the same serving target. That has direct consequences for GPU infrastructure cost optimization, where serving-layer efficiency gains compound across the scale of an inference fleet.

DOP is worth attention beyond the switching context. It eliminates per-rank weight-replication overhead, freeing 12.69 GiB per GPU at T=8 for KV cache. For teams whose limiting resource is KV capacity rather than compute, DOP as a fixed layout on long-context workloads is a meaningful option independent of SPLASH’s dynamic switching.

The broader design lesson connects to how modern agent serving stack design must account for dynamic load. Agentic workloads do not arrive as stationary traffic distributions. A batch of agent episodes will have a wide spawn phase, a middle phase where some agents hit dead ends and terminate, and a tail phase where the survivors run long multi-step traces. That profile is structurally similar to an RL rollout and faces the same problem: a layout that serves the spawn phase poorly serves the tail phase and vice versa. SPLASH’s scheduler responds to the shape of the live batch rather than the expected shape at deployment, which is the right level of abstraction for workloads the operator cannot fully characterize in advance.

Key Takeaways

• SPLASH (Chuan Liu et al., Institute of Computing Technology, Chinese Academy of Sciences, arXiv:2609.37626, September 2026) switches LLM attention parallelism layout during live inference without draining the running batch. Median overhead is under 0.51% of the serving step in which the switch occurs; end-to-end switch time ranges from 0.02 to 11.76 ms versus up to 667.31 ms for a blocking restart.
• The enabling fact is that GQA, MQA, and MLA decouple the KV cache from the weight sharding dimension. Weight layout and KV ownership can be redistributed independently. SPLASH is not applicable to classic multi-head attention models where those two choices are coupled.
• SPLASH introduces Decoupled Ownership Parallelism (DOP), a fourth layout where both weights and KV scale with rank count: M_DOP(s) = W_A/T + B*k*s/T. DOP frees 12.69 GiB per GPU at T=8 (FP8) compared to DP-attention by eliminating per-rank weight replicas.
• Throughput improves 1.3 to 1.73 times over fixed-layout deployments on B200 GPUs serving GLM-5.3, with the same layout-transition pattern appearing on DeepSeek-V3.2 (H200) and GLM-5.3-Flash (DCU). The system is most applicable to workloads where batch composition shifts from wide-and-short to narrow-and-long during execution.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If your LLM serving infrastructure needs to handle mixed workloads efficiently, including agentic pipelines, RL rollouts, or any scenario where batch shape changes mid-execution and a fixed parallelism layout is leaving throughput on the table, talk to our team.