Seven Architectures Replacing KV Cache With Bounded Mutable State

On September 21, 2026, a team from the University of Freiburg, Microsoft Research, and affiliated labs submitted Complex KDA (arXiv:2609.24797), extending Kimi Delta Attention with complex-valued transitions that reach the state-tracking expressivity of DeltaProduct-2 inside a single recurrent update. The same week, ONNX opset 27 shipped a LinearAttention operator with past_state and present_state as first-class inputs and outputs, codifying recurrent-state semantics in the same interchange format that governs how models move between training clusters, inference servers, and edge runtimes. Together these two events make one thing legible: the inference session is no longer defined by a growing collection of per-token keys and values. It is defined by a bounded mutable matrix that the model edits token by token.

This article compares seven architectures that treat context as a fixed-size mutable state object rather than an ever-expanding cache. The comparison spans the full production maturity range, from Kimi Linear’s deployed hybrid model with open vLLM kernels, to Kalman Delta Networks’ September 2026 research preprint that adds uncertainty tracking to recurrent associative memory. Ordering runs from most to least production maturity; the rationale is explained in “What This Category Reveals.”

Why This Comparison Is Timely

Three overlapping events in 2026 crystallized the question. First, Moonshot AI’s Kimi Linear (arXiv:2510.26692, released October 2025 and deployed at scale by 2026) demonstrated that a hybrid model with 75% KDA layers and 25% full-attention layers outperforms full MLA across short-context, long-context, and reinforcement-learning evaluations at 1.4T training tokens, while cutting KV cache usage by 75% and reaching 6.3x faster time-per-output-token at 1M context length. Production systems at that scale do not ship without confidence in the approach.

Second, NVIDIA submitted Gated DeltaNet-2 (arXiv:2605.22791, May 21, 2026) from researchers Ali Hatamizadeh, Yejin Choi, and Jan Kautz. At 1.3B parameters trained on 100B FineWeb-Edu tokens, it topped Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across language modeling, commonsense reasoning, and RULER retrieval. The paper ships working kernels and a GitHub repository, not just a theory result.

Third, ONNX opset 27 defines LinearAttention with an update_rule attribute selecting among “linear,” “gated,” “delta,” and “gated_delta” recurrences, and with past_state / present_state tensors of shape (B, H_kv, d_k, d_v). Serving runtimes that handle ONNX models now have to reason about transferring, quantizing, and checkpointing this fixed-size matrix, not a ragged cache. The engineering problem is no longer hypothetical.

The Shift: Growing Cache to Bounded Mutable State

Standard softmax attention stores one key vector and one value vector per token per layer. At 128k tokens, a 70B model with 80 layers runs a KV cache large enough to saturate a serving cluster’s memory bandwidth on every decode step, regardless of batch size. The compute required to score a new query against all past keys grows linearly in sequence length and translates directly into latency and cost at scale.

Linear recurrent attention replaces that growing cache with a matrix state S_t of fixed dimension d_k x d_v. Every new token updates S_t using a recurrence rule, and the query reads from S_t alone. Memory consumption during decode does not grow with context length. The trade-off is compression: S_t must summarize potentially millions of tokens into a matrix whose size does not depend on sequence length. Earlier associations can be overwritten or decay. The architectures compared here differ primarily in how they control that overwriting, how they allocate state capacity across key dimensions, and what secondary mechanisms they add to recover precision that pure compression loses. The question every team is answering is the same: given a fixed budget of state dimensions, how do you decide what to keep, what to erase, and what to write?

Kimi Linear (Kimi Team, arXiv:2510.26692): Channel-wise Gating at Scale

Kimi Linear is the most production-validated architecture in this comparison. The Kimi Team (Moonshot AI) published the technical report in October 2025 and has since deployed the 48B-parameter Kimi-Linear-48B-A3B-Instruct checkpoint, released on Hugging Face with open vLLM kernels. The model uses a 3:1 ratio of KDA layers to full Multi-Head Latent Attention (MLA) layers. That ratio was not picked arbitrarily: the team ran controlled scaling-law experiments and found that one full-attention layer per four KDA layers retains global information flow while cutting KV cache usage by 75% compared to pure MLA.

The mechanism at the core is Kimi Delta Attention. KDA extends Gated DeltaNet by replacing its scalar per-head forget gate with a channel-wise gate alpha_t in R^{d_k}. Each key-dimension channel maintains its own decay rate, so a single attention head can simultaneously preserve long-lived associations in some channels and rapidly forget transient ones in others. The state update becomes S_t = Diag(alpha_t) * S_{t-1} + beta_t * k_t * (v_t - Diag(alpha_t) * S_{t-1}^T * k_t)^T, a rank-one correction to the decayed state. The DPLR (Diagonal-Plus-Low-Rank) transition structure enables a specialized chunkwise-parallel kernel that runs substantially faster than the general DPLR formulation while remaining consistent with the classical delta rule.

Empirically: at 1.4T training tokens with 1.4B activated parameters (3B total), Kimi Linear scores 51.0 on MMLU-Pro (4k context) versus 47.2 for full MLA, and 84.3 on RULER (128k context) versus 81.3 for MLA, while achieving 3.98x decoding acceleration on the RULER task. At 1M context length, time-per-output-token is 1.84ms versus 11.48ms for MLA, a 6.3x difference. The team also open-sourced the KDA kernel and vLLM integration at flash-linear-attention, making KDA a drop-in for existing pipelines without cache interface changes. The main limitation is that purely linear-structure layers are still constrained by finite-state capacity for multi-item associative recall; RULER multi-needle tasks expose this, which is why the architecture retains full attention layers rather than removing them.

Gated DeltaNet-2 (NVIDIA, arXiv:2605.22791): Decoupled Erase and Write Gates

NVIDIA’s Gated DeltaNet-2, submitted May 21, 2026 by Ali Hatamizadeh, Yejin Choi, and Jan Kautz, addresses a specific bottleneck in all prior delta-rule models: the scalar tie between erasing and writing. In Gated DeltaNet and KDA, a single scalar gate beta_t controls both how much old content to erase on the key side and how much new content to commit on the value side. These are different operations acting on different axes of the state matrix. Erasing is key-side: it decides which coordinates of the existing read to remove. Writing is value-side: it decides which coordinates of the incoming value to commit. Conflating them with one scalar prevents the model from, for example, erasing selectively without also weakening the write, or writing strongly without also over-erasing.

GDN-2 separates these with a channel-wise erase gate b_t and a channel-wise write gate w_t. The recurrence becomes S_t = Diag(alpha_t) * (I - b_t * k_t * k_t^T) * S_{t-1} + w_t * k_t * v_t^T, where b_t and w_t are channel-wise vectors rather than scalars. Setting b_t = w_t = beta_t * 1 recovers KDA exactly; further tying decay to a scalar recovers Gated DeltaNet. The paper derives a chunkwise WY algorithm that absorbs cumulative channel-wise decay into asymmetric erase factors, preserving efficient parallel training with a gate-aware backward pass.

Benchmark results at 1.3B parameters on 100B FineWeb-Edu tokens place GDN-2 at the top of the evaluated field across language modeling, commonsense reasoning, and RULER needle-in-a-haystack retrieval, with the most pronounced gains on multi-key RULER tasks where a fixed-size state must separate competing associations. Code and kernels are open at NVlabs/GatedDeltaNet-2. As of this writing, GDN-2 has not appeared in a deployed product at the scale of Kimi Linear, but its hardware-efficient kernel and the NVIDIA provenance suggest production adoption is close. The architecture is also the genealogical parent referenced by Complex KDA, confirming its centrality to the design space.

Mamba-3 (Carnegie Mellon University and Princeton, arXiv:2603.15569): Complex SSM State with MIMO Decode

Mamba-3, submitted March 16, 2026 by Aakash Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu at CMU, Princeton, Together AI, and Cartesia AI, extends the SSM lineage rather than the delta-rule lineage. Where KDA and GDN-2 work in the fast-weight / linear-attention formulation, Mamba-3 derives its improvements from state-space model discretization theory. Three mechanisms combine into the final architecture.

Exponential-Trapezoidal discretization replaces the Euler-based discretization used by Mamba-1 and Mamba-2. The trapezoidal method produces a more expressive recurrence that can be expanded into an implicit convolution, which means Mamba-3 can replace the short causal convolution previously considered essential for recurrent models. Complex-valued state transitions give each SSM state element a complex number rather than a real one. The paper shows this is equivalent to a data-dependent rotary embedding, and that the complex update solves synthetic state-tracking tasks (including parity of bit sequences) that Mamba-2 cannot solve at all. Multi-Input, Multi-Output (MIMO) formulation replaces the outer-product state update with a matrix-multiplication state update. This raises arithmetic intensity during decode without increasing state size, converting a memory-bound decode step into one that better uses GPU tensor cores.

Quantitative results at 1.5B scale: Mamba-3 (MIMO) improves downstream accuracy by 2.2 points over Transformers, 1.9 points over Mamba-2, and 1.8 points over Gated DeltaNet. Mamba-3 (MIMO) with state size 64 matches Mamba-2 with state size 128 on perplexity, halving the state size at equal quality. Across state-size experiments, Mamba-3 (SISO) improves over the next-best model, GDN, by 0.6 points. Fast training and inference kernels are released at the state-spaces/mamba repository. The retrieval ceiling remains a limitation: the paper notes that retrieval on tasks requiring multi-item associative recall still improves with hybrid attention layers, and Mamba-3 results for retrieval are shown alongside hybrid configurations.

HOLA (Wanyun Cui / Shanghai University of Finance and Economics, arXiv:2607.02303): Bounded Exact Cache as Hippocampal Complement

HOLA (Hippocampal Linear Attention), submitted July 2, 2026 by Wanyun Cui at Shanghai University of Finance and Economics, takes a different path. Rather than improving the recurrent state’s ability to compress information, HOLA accepts that compression is lossy by design and adds a bounded exact KV cache alongside the recurrent state. The design is inspired by Complementary Learning Systems (CLS) theory from neuroscience: the neocortex compresses slowly across experiences, while the hippocampus records specific novel events for exact recall. HOLA implements this as a semiparametric test-time memory where the delta-rule recurrent state is the parametric estimator for linearly compressible structure, and a bounded exact cache is the non-parametric correction for associations the state cannot absorb.

The key design question is eviction policy: which tokens does the bounded cache keep? Sliding-window approaches (used in most hybrid linear-attention models) keep the most recent tokens, but a fact that is distant and must be recalled exactly disappears once it slides out of the window. HOLA uses intrinsic write magnitude as the eviction score: the delta-rule model already computes how surprising each token is to its current state (the residual beta * ||e||). Tokens with large write magnitude changed the state most and are therefore the ones the non-parametric cache should retain. A matched control at 340M parameters confirms that importance-based eviction beats recency-based eviction on perplexity and long-context retrieval, while commonsense reasoning stays within single-seed noise.

At 340M parameters trained on 15B SlimPajama tokens, HOLA reduces WikiText perplexity from 27.32 to 22.92 (a 16.1% reduction), which falls below full-attention Transformer++ at 26.88. LAMBADA perplexity improves from 30.95 to 30.26. On RULER S-NIAH-1 needle-in-a-haystack recall, HOLA remains stronger than GDN and HOLA+recency as context grows to 32k tokens, which is 16x its training length. The paper notes that a decoupled RMSNorm-gamma on the cache read path (Qwen3-style) is required to restore sharp retrieval: without it, cache softmax becomes nearly uniform and the exact memory degenerates into a lossy average. The limitation is that HOLA adds a bounded exact cache on top of a recurrent state, so its snapshot size for speculative decoding includes both components. Cache size tuning is a hyperparameter not fully explored at scale above 340M.

Complex KDA (Siems et al., arXiv:2609.24797): Orthogonal State Transitions for Richer Expressivity

Complex KDA (CKDA), submitted September 21, 2026 by Julien Siems, Riccardo Grazzi, Korbinian Pöppel, and collaborators from the University of Freiburg, Microsoft Research, the University of Tübingen, EPFL, and the Jülich Supercomputing Center, extends KDA’s channel-wise gating with complex-valued parameter ranges that enable 2D rotations within a single recurrent update. The core observation is that prior delta-rule models with diagonal-plus-rank-one transitions can model only Householder reflections in each update step. Composing two delta-rule transitions can model a 2D rotation, but that doubles rank and cost. CKDA gets rotations from a single transition by combining the delta-rule Householder reflection with a sign flip from the channel-wise gate (setting gate range to [-1, 1]) and allowing the delta coefficient beta in [0, 2].

The mathematical result: every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of SO(3), and many state-tracking proofs use one fewer layer for CKDA compared to other diagonal-plus-rank-one linear RNNs. The transitions remain non-expansive (operator norm stays at most 1), preserving stability. The paper demonstrates that combining both parameter range extensions yields the strongest length extrapolation on S3, S4 group-word tracking and periodic audio continuation benchmarks. In language modeling, CKDA outperforms Transformers and other linear RNNs, and shows scaling behavior comparable to a KDA baseline while adding the expressivity of 2D rotations. Code and pretrained models are open-source per the paper’s abstract.

What CKDA does not claim: the expressivity gains in state tracking do not automatically translate to proportional language modeling gains over a well-tuned KDA baseline. The paper is explicit that language-modeling results are “similar to a KDA baseline,” with the gains concentrated in the structured expressivity benchmarks. For a serving team, this means CKDA is a research-stage extension that improves theoretical guarantees and specific structured tasks, not a drop-in production replacement for KDA yet. The submission date of September 21, 2026 makes it the freshest entry in this comparison.

Kalman Delta Networks (Bui, Huang, Ying / Yale University, arXiv:2609.07816): Uncertainty-Aware Memory Updates

Kalman Delta Networks (KDNs), submitted September 7, 2026 by Ngoc Bui, Tinglin Huang, and Rex Ying at Yale University’s Department of Computer Science, reframes the recurrent associative memory problem as linear-Gaussian state-space estimation. The motivation is precise: delta-rule models predict write gain from the current token embedding, but they do not track confidence in the stored association. A large residual (v_t – S_{t-1}^T k_t) might mean the stored association is stale and should be revised aggressively, or it might mean this is a noisy observation of an already well-established association that should be discounted. Without uncertainty tracking, the model cannot distinguish these cases.

KDN models the recurrent memory as an estimate of a latent, non-stationary key-value map. Each token supplies a noisy observation of that map at one key, while the learned transition describes how the map persists or drifts between observations. Under linear-Gaussian assumptions, the Kalman filter is the optimal recursive estimator: it propagates both the memory estimate and its covariance, and assigns each residual write a gain that balances memory uncertainty against observation noise. DeltaNet, Gated DeltaNet, and KDA emerge as covariance-free approximations that omit this uncertainty recursion. Exact Kalman updates are impractical (dense d_k x d_k covariance per head, state-dependent Riccati recurrence), so the paper introduces two scan-compatible approximations. Diagonal KDN uses mean-field variational inference to project the posterior back onto the diagonal family, adding O(d_k) auxiliary state per head. Isotropic KDN uses a single scalar uncertainty per head, adding O(1) auxiliary state. Both uncertainty recurrences are Möbius maps, enabling associative scans with logarithmic parallel depth.

Results from controlled pretraining on FineWeb-Edu at 750M/50B and 1.3B/100B parameter-token settings: both KDN variants achieve lower WikiText and LAMBADA perplexity and higher mean six-task zero-shot accuracy than every evaluated linear-time recurrent mixer including Mamba-3, KDA, and GDN-2 at both scales. Diagonal KDN achieves the highest observed 14-cell RULER aggregate at both scales. The limitation the paper acknowledges is the diagonal approximation: mean-field projection can underprotect stored key directions, which the authors address with an information-scaling factor but cannot eliminate entirely. KDNs also add auxiliary state that must be managed, quantized, and checkpointed alongside the main recurrent state.

ONNX LinearAttention / FlashInfer (ONNX Opset 27): Runtime Standardization of Recurrent State Semantics

The LinearAttention operator in ONNX opset 27 is the most concrete signal that recurrent state is becoming a runtime primitive rather than a research artifact. The operator defines past_state and present_state as optional 4D tensors with shape (B, H_kv, d_k, d_v), where B is batch size and H_kv is the number of key-value heads. The update_rule attribute accepts four modes: “linear” (basic additive), “gated” (exponential decay), “delta” (delta rule without decay), and “gated_delta” (the full Gated DeltaNet / KDA family). The default is “gated_delta.” Group-query attention is supported through a q_num_heads / kv_num_heads ratio, with each KV head and its associated recurrent state shared across grouped query heads.

What this means operationally: any exporter that targets ONNX opset 27 must now produce a model graph where the recurrent state is an explicit tensor that flows between inference steps, exactly like the KV cache tensors flow in transformer ONNX exports. Serving runtimes that consume ONNX graphs (ONNX Runtime, Triton Inference Server via ONNX backend, and edge deployment toolchains) must allocate, transfer, and manage this state tensor. The fixed shape of the state — it does not grow with sequence length — is the key difference from KV cache management. For a 48B model with 64 heads, d_k=128, d_v=128, at float16, the recurrent state per request is 64 * 128 * 128 * 2 bytes = 2 MB, constant regardless of context length. A KV cache for the same model at 128k tokens is on the order of 64 GB.

FlashInfer, the high-performance attention kernel library used by vLLM and SGLang, has added corresponding linear attention support. The engineering consequence for serving teams is that state snapshots now need to be saved and restored for speculative decoding rollback, migrated between workers for request routing, and quantized for memory efficiency. DAMP (arXiv:2608.27513), submitted August 27, 2026 by Tao Zhang, Jianchao Tan, and collaborators at Meituan and South China University of Technology, addresses exactly this: post-training quantization of GDN and KDA recurrent states in production models. DAMP finds that uniform INT8 and FP8 already degrade reasoning accuracy, while INT4 and NVFP4 reduce it near zero. At 9.9 bits per state value (mixed precision with high-risk channels in higher precision), DAMP reduces recurrent state storage by 69.1%, accelerates the state update kernel by up to 2.01x, and lowers full-model time-per-output-token by up to 10.9% on Qwen3.6-35B and Kimi-Linear-48B across six benchmarks.

How They Compare

What This Category Reveals

The ordering in this article runs from most production-mature to most research-stage. Kimi Linear leads because it is a deployed 48B model with open vLLM kernels verified across 1.4T training tokens. Gated DeltaNet-2 follows because NVIDIA shipped working CUDA kernels alongside the theory paper. Mamba-3 follows because it has CMU/Princeton/Cartesia backing and public kernels, but no deployed product at the scale of Kimi Linear. HOLA, Complex KDA, and Kalman Delta Networks are pure research contributions at the time of writing, ordered by the novelty and scope of the mechanism they introduce: HOLA adds a qualitatively new component (bounded exact cache); Complex KDA deepens expressivity theory; Kalman Delta Networks introduces a new statistical framing. ONNX LinearAttention is last not because it is immature but because it is infrastructure rather than a model architecture; it belongs in the comparison because it makes the other entries deployable across runtimes.

What the category reveals as a whole: the key differentiating factor between the entries is not raw state size but state update quality. Kimi Linear and GDN-2 both use fixed-size states with similar dimensions, but GDN-2’s decoupled erase/write gates give it a cleaner editing interface. Kalman Delta Networks add uncertainty tracking that no prior delta-rule model had. HOLA adds exact recall for the tokens that mattered most. Complex KDA adds the ability to represent rotations (not just reflections) in a single recurrent step. The open engineering question the category has not answered: what happens to state quality when you combine uncertainty tracking (KDN), importance-based exact backup (HOLA), and complex-valued transitions (Mamba-3, Complex KDA) in one architecture? Each paper improves one dimension. None combines all four.

Limitations and Open Questions

Fixed-size recurrent memory is compressive by design. When a model processes a 128k-token document, the state at token 128,000 must encode everything useful from all 128,000 previous tokens. It cannot. Associations from early tokens decay or are overwritten by later writes, which is why all deployed hybrid architectures retain some full-attention layers. The RULER needle-in-a-haystack benchmarks expose this directly: multi-key retrieval accuracy drops with state-only architectures in ways that hybrid models only partially recover.

State rollback for speculative decoding is a real engineering problem that the research papers do not address. Tree speculative decoding (as in GDN Tree-Scan, arXiv:2609.23900) must carry the recurrent state that sequential decode would have produced along each branch path. For attention-only transformers, the verifier needs only an ancestry mask. For recurrent-hybrid models, it needs branch-local state scan/replay, which adds implementation complexity and latency overhead. GDN Tree-Scan reports a 27.0% token-weighted decode-throughput gain at batch-1 with a six-node tree on Qwen3.6-27B-FP8, but the implementation is non-trivial.

State quantization matters more than KV cache quantization did. DAMP (arXiv:2608.27513) found that INT8 state quantization already degrades reasoning accuracy on Qwen3.6-35B and Kimi-Linear-48B, while INT4 collapses it. This is a qualitatively different failure mode from KV cache quantization, where INT8 is generally safe. The reason is error propagation: recurrent state errors accumulate over a sequence, while KV cache errors are bounded to the attention heads that read a particular cached vector. At 9.9 bits per state value, DAMP recovers near-FP32 accuracy with a 69.1% reduction in state storage. The field does not yet have a standard quantization specification for recurrent states, analogous to the INT8/FP8 KV cache standards in transformer serving.

State migration between workers during continuous batching has no established protocol. In transformer serving, KV cache can be migrated by copying a ragged buffer. For recurrent models, the state is fixed-size and simpler to copy, but the serving software must track state per active request and preserve it across token steps without the implicit buffer management that Paged Attention provides for KV caches. vLLM and SGLang have begun adding this, but it is not yet at parity with transformer KV cache management maturity.

What This Means for Engineering Teams

If your team runs inference on 128k+ context requests at scale, the memory arithmetic alone justifies evaluating Kimi Linear or a GDN-2-based hybrid. A constant 2 MB recurrent state per request versus a 64 GB KV cache for 128k tokens at 48B scale is not a marginal difference. The latency implications for batching and memory bandwidth are proportional. The practical starting point is the Kimi-Linear-48B-A3B-Instruct checkpoint, which ships with vLLM integration and has been evaluated across standard reasoning benchmarks.

For teams building new model architectures rather than deploying existing ones, the design decision with the most downstream consequences is whether to use a state-only recurrent architecture or a hybrid. The evidence from every deployed model in this comparison (Kimi Linear, and production GDN hybrids like Qwen3-Next with 75% GDN layers) points strongly toward hybrid: one full-attention layer per three or four recurrent layers appears to be the current Pareto point for preserving exact recall capability while capturing most of the efficiency benefit. Pure recurrent architectures remain weaker on multi-key retrieval benchmarks.

ONNX opset 27 is the convergence point for tooling. If your serving pipeline uses ONNX Runtime or any ONNX-compatible backend, the LinearAttention operator with past_state / present_state is how recurrent-state models will be expressed at the interchange layer. Adapting your inference pipeline to pass state tensors between decode steps — rather than assuming a stateless forward pass per request — is the infrastructure work to prioritize. Teams working on production LLM inference will find useful context in the Origins AI analysis of LLM inference fundamentals and memory optimization for transformer workloads.

State quantization is an immediate practical concern for anyone deploying KDA or GDN-based models at batch size greater than 32. DAMP (arXiv:2608.27513) provides a calibration-based offline protocol: identify high-risk key channels using accumulated-error energy and decay-based persistence, store those channels at higher precision, and store the remainder at INT8. The 2.01x kernel speedup and 10.9% TPOT reduction at 9.9 bits per value are reproducible targets. Budget calibration time (approximately 100-200 forward passes on representative prompts) before deploying quantized states.

Key Takeaways

• Kimi Linear’s 48B model achieves 51.0 on MMLU-Pro and 84.3 on RULER-128k, outperforming full MLA at identical training compute, while cutting KV cache usage by 75% and reaching 6.3x faster decode at 1M context.
• Gated DeltaNet-2 (NVIDIA, May 2026) shows that decoupling erase and write into separate channel-wise gates, rather than one scalar, produces the strongest retrieval results among evaluated 1.3B recurrent models at 100B training tokens.
• Mamba-3’s complex-valued SSM state solves synthetic state-tracking tasks (parity, group-word problems) that Mamba-2 cannot, while the MIMO formulation achieves Mamba-2 perplexity at half the state size.
• HOLA reduces WikiText perplexity to 22.92 at 340M parameters (below full-attention Transformer++ at 26.88) by pairing a delta-rule recurrent state with a bounded exact cache that retains high-surprise tokens rather than recent ones.
• DAMP’s post-training quantization of recurrent states at 9.9 bits per value maintains near-FP32 accuracy while reducing state storage by 69.1% and accelerating the state update kernel by up to 2.01x, with uniform INT8 already degrading reasoning accuracy on complex benchmarks.
• ONNX opset 27’s LinearAttention operator with past_state / present_state tensors standardizes recurrent-state semantics at the interchange layer, making state management a first-class responsibility for serving runtimes.
• No published architecture yet combines uncertainty-aware updates (Kalman Delta Networks), importance-based exact backup (HOLA), complex-valued transitions (Mamba-3/CKDA), and decoupled erase/write (GDN-2) in one model; each paper improves one dimension independently.

Work With Origins AI

Origins AI builds production AI infrastructure for engineering teams. If your team is evaluating recurrent-state architectures for long-context inference or navigating the transition from KV-cache-based serving to fixed-state hybrid models, talk to our team.