Seven AI Systems That Now Optimize the Production Inference Stack

On September 15, 2026, researchers from Unity published FlashVector, a framework that sent AI agents into the C++ codebase of an NVIDIA Triton model server and the Python feature-processing service of a live advertising platform, then measured the results: up to 2x throughput on the model server, up to 1.6x throughput on the feature store. Two days later, Z.ai described how a GLM-5.3-powered infrastructure agent helped bring a production inference stack online across more than 100,000 domestic AI accelerators in under two weeks, achieving roughly 3x end-to-end throughput. Both accounts describe the same structural shift: AI agents are no longer writing application code and moving on. They are closing performance feedback loops inside the infrastructure itself.

Why This Comparison Is Timely

The FlashVector paper (arXiv:2609.17391, September 15, 2026) and Z.ai’s production account (September 17, 2026) make a useful pair. FlashVector shows the breadth of the optimization target, extending the kernel-generation paradigm into model-server C++ and Python feature pipelines. Z.ai shows the scale: a fully deployed, continuously tuned system at 100,000-accelerator size. Together they advance the claim from “AI can generate GPU kernels” to “AI can function as the performance engineering team for a production serving stack.”

That shift has support from a cluster of concurrent research. AMD published GEAK v4 results in July 2026 showing agents profiling SGLang and vLLM workloads and improving serving throughput by 60%, 96%, and 42.2% on three evaluated configurations. Alibaba open-sourced the Atrex Kernel Agent with support for NVIDIA, AMD, and T-Head hardware. WarpSpeed from doubleAI beat NVIDIA’s optimized PyTorch baselines on 90% of 235 Blackwell benchmarks after a single day of autonomous search. What was a collection of isolated kernel-generation studies in early 2026 is now a recognizable infrastructure category.

The Shift: Manual Kernel Tuning Becomes a Closed Loop

The traditional approach to GPU inference performance requires a systems engineer who understands the specific hardware, the kernel language, and the workload. That engineer profiles a bottleneck, writes a candidate kernel, benchmarks it against the baseline, rejects it if it is wrong or slower, and iterates. Across a serving stack with multiple layers, from CUDA kernels through computation graphs through model-server C++ through feature-processing Python, this process takes weeks per component and demands rare expertise at each layer.

The systems described here replace the human iteration with an agent-driven loop. An agent receives the workload, identifies bottlenecks by running profilers, generates candidate code, submits it to a correctness gate, benchmarks it against the existing baseline, retains improvements, and continues. The human role shifts from writing kernels to defining objectives, setting correctness contracts, and approving production changes. What this makes possible: the loop can run continuously, across hardware targets that lack hand-tuned kernels, at optimization depths that would be impractical for a human team. Systems engineering turns out to be unusually well suited to this pattern because performance is measurable and candidate changes can be accepted or rejected automatically.

FlashVector: Extending Kernel Agents to the Full Serving Stack

FlashVector, submitted September 15, 2026 (arXiv:2609.17391), describes what happens when the scope of kernel optimization expands to include every layer above the GPU. The paper identifies model serving in production recommender systems as “one of the largest cost drivers,” and notes that “automated tuning for the broader serving stack remains largely unexplored.” Its response is an extensible framework that applies optimization agents not just to GPU kernels but to ML framework computation graphs, the C++ codebase of an NVIDIA Triton model server, and a Python on-demand feature transformation service.

The architecture is hierarchical. Agents at each layer receive profiling feedback specific to that layer and generate candidates constrained by that layer’s semantics. A kernel agent does not write model-server C++ code, and a model-server agent does not generate CUDA. This separation keeps the search space tractable at each level while allowing the overall optimization to propagate improvements across all four layers. FlashVector describes this as an “extensible framework to generalize the single kernel optimization agent paradigm to heterogeneous technical stacks, and to deliver performance improvements holistically.”

In production at Unity’s Vector advertising platform, FlashVector delivered up to 2x throughput improvement on the model server, up to 1.98x latency improvement on the model server, and up to 1.6x throughput improvement on the feature store. These are the broadest scope results in this comparison. The paper’s contribution is not a new kernel language or a faster profiler; it is the argument that single-layer kernel optimization is an unnecessarily narrow target when the bottleneck may be anywhere in the serving hierarchy.

Z.ai GLM Infra Agent: 100,000 Accelerators in Two Weeks

Z.ai’s September 17, 2026 account of building the inference infrastructure for GLM-5.3-Flash describes what closed-loop infrastructure optimization looks like in practice at scale. A GLM-5.3-powered infrastructure agent participated in bringing a production serving stack online across more than 100,000 domestic AI accelerators. The team reached production in under two weeks. End-to-end throughput reached approximately 3x compared to the initial configuration.

The Z.ai account is notable for what it says about the human role. Engineers retained responsibility for defining objectives, ensuring numerical correctness of results, managing concurrency risk, and making production-approval decisions. The agent operated inside those constraints, working across kernel optimization, runtime debugging, and concurrency analysis rather than application-level code generation. The technical blog includes concrete examples of kernel profiling and runtime-event analysis, which distinguishes it from high-level vendor framing.

The scale is the data point that matters most here. Most kernel optimization research reports results on a single machine or small cluster. Z.ai’s account involves 100,000 accelerators and a two-week bring-up timeline. If accurate, it suggests that agentic infrastructure tooling can compress hardware bring-up timelines significantly. That matters most for operators deploying on non-CUDA accelerators where hand-tuned kernels do not already exist. Instead of waiting months for a hardware vendor’s kernel team to produce optimized code, an operator can bring a profiler, a correctness harness, and an optimization agent.

AMD GEAK v4: Profiling SGLang and vLLM to Target End-to-End Serving Throughput

GEAK (GPU Efficiency and Automation Kit) v4, published by AMD in July 2026, extends the kernel optimization paradigm from individual kernels to end-to-end serving workloads. Where earlier GEAK versions optimized individual kernel files, v4 profiles running SGLang and vLLM deployments, ranks identified bottlenecks by their end-to-end impact on serving throughput, and recursively invokes kernel agents on the highest-impact targets. Results are validated on real AMD hardware before acceptance.

GEAK v3 provides the closest benchmark context. It achieved a 3.02x geometric-mean speedup on HIP kernels across 16 evaluated kernels, with L1 kernels reaching 4.14x and L2 kernels reaching 4.20x. On Triton kernels across 17 evaluated examples, the geometric mean was 2.22x. Production case studies include a DeepSeek V4 MLA kernel that improved end-to-end throughput by 2.10x and reduced time-to-first-token by 3.71x, and a GPT-OSS 120B deployment that saw 8-10% uplift from attention kernel optimization. Hardware targets span AMD Instinct CDNA (MI300X, MI355X) and Radeon RDNA4 (gfx1201, Navi 48).

GEAK v4 moves the objective function from kernel-level speedup to serving-level throughput. On three evaluated serving configurations, it reported improvements of 60%, 96%, and 42.2%. The 96% result in particular represents a configuration where the agent identified a bottleneck that did not manifest in isolated kernel benchmarks. This is the key advantage of profiling at the serving level: bottlenecks visible only under real request distributions become addressable. A kernel that looks optimal in isolation may be the wrong kernel for the actual request pattern hitting the serving engine.

Alibaba Atrex Kernel Agent: Open-Source, Multi-Hardware, Profile-Driven

The Atrex Kernel Agent (AKA), open-sourced by Alibaba, takes a different approach to the same problem. Rather than building a proprietary optimization system, Alibaba published the full orchestration framework with support for NVIDIA, AMD, and T-Head PPU (zwm890p) hardware and five kernel languages: Triton, CuteDSL, CUDA, FlyDSL, and TileLang. The system runs through a single entry point, orchestrator/optimize.py, that drives an episode-based optimization workflow: generate, profile, check correctness, record history, iterate.

Two design choices stand out. First, Atrex uses Git-isolated optimization branches with a canonical measurement history. Each optimization episode creates a branch, generates candidate code, benchmarks it against the canonical baseline, and only merges changes that pass both correctness and performance gates. This makes the optimization resumable: if a remote GPU server fails mid-episode, the system recovers to the last stable state. Second, the system supports multiple AI backends including Claude, Qoder, Codex, and Pi, which means the optimization agent can be swapped without changing the surrounding infrastructure.

Atrex does not report a single aggregate benchmark number. Its validation came from a production competition: in July 2026, the system helped Qwen3.8 achieve the top ranking on the SOL-ExecBench FlashInfer operator optimization leaderboard. The system supports both leaderboard mode and fail-closed production mode. In fail-closed mode, a change that does not beat the baseline by a defined margin is rejected rather than flagged for human review. That distinction matters for production use: fail-closed behavior means no regression can pass the gate undetected, which is a stronger correctness guarantee than a system that defers borderline cases to a human reviewer.

WarpSpeed: Autonomous Search at the Speed of Blackwell

WarpSpeed, from doubleAI, takes the most aggressive stance on automation in this comparison. The system designs, implements, verifies, specializes, and tunes kernels for a target hardware platform through what the team calls “artificial expert intelligence” for performance engineering. On the SOL-ExecBench benchmark suite, covering 235 Blackwell GPU kernels across four problem categories (atomic single-operation kernels, fused multi-operation blocks, quantization kernels, and FlashInfer-Bench inference primitives), WarpSpeed beat NVIDIA’s optimized PyTorch baselines on 90% of kernels, with a geometric-mean speedup of 2.24x across all benchmarks. The entire optimization run completed within a single day.

The performance breakdown by category shows where autonomous search delivers the most. On quantization kernels, WarpSpeed delivered its largest gains: an NVFP4 grouped-query attention kernel ran 14.9x faster than the optimized reference. A useful comparison point is Cursor’s multi-agent system, which ran for three weeks on the same benchmark and achieved a 63% win rate with a 1.38x average speedup. WarpSpeed won across all four categories in one day against a three-week run by a competing system.

WarpSpeed’s research also surfaces a significant concern about the broader kernel optimization benchmark landscape. The team found that standard verifiers, including SOL-ExecBench’s own verification framework, permit several failure modes: precision loss bugs that pass verification under uniform input distributions but fail under Zipfian distributions, seed overfitting (eight previously-passing kernels failed with fresh random seeds), and hardcoded solutions that encode precomputed constants for specific parameter values. WarpSpeed built a custom verification framework to address these cases. The implication for the field: reported benchmark wins for kernel optimization agents may be inflated if the verifier does not exercise realistic input distributions.

AgentCompile: Compiler and LLM Working Together

AgentCompile (arXiv:2606.07665, June 2026) takes a different structural approach from the profiling-and-iteration systems above. Instead of an agent that generates arbitrary kernel code and validates the output, AgentCompile uses two complementary strategies. In the first, the LLM supplies advisory metadata for region summaries and bounded candidate spaces; the compiler then instantiates template-based CUDA candidates, validates correctness, selects implementations by measured latency, and falls back to a known-good baseline if no candidate improves on it. In the second, the LLM directly generates five classes of decode-critical kernels under distilled optimization principles, working within contracts defined by the compiler rather than exploring an open-ended search space.

The compiler-bounded approach addresses a fundamental problem that WarpSpeed’s correctness research surfaces. As the AgentCompile paper states: “unconstrained generation guarantees neither correctness nor performance.” By having the compiler constrain the candidate space, the system avoids seed overfitting and distribution exploitation at the cost of some search flexibility. The fallback mechanism ensures that a failed optimization attempt leaves performance unchanged rather than regressing it. This conservative design makes AgentCompile’s results less dramatic than WarpSpeed’s but more predictable in production.

AgentCompile includes a complete serving runtime: paged KV cache, continuous batching, preemption, chunked prefill, and bucketed CUDA Graph replay. Benchmark results cover six model families. Against PyTorch eager execution, the system delivers 2.23-6.98x speedups for single-request generation. Against vLLM, which already incorporates substantial optimization, it achieves 1.04-1.16x improvement for both single-request and multi-request serving. The modest gains over vLLM are honest: a 4-16% improvement over an already-optimized serving engine represents real additional throughput at a production operating point.

FlashInfer-Bench: The Deployment Loop That Makes Kernel Substitution Safe

FlashInfer-Bench (arXiv:2601.00227, January 2026) is the entry in this comparison least like the others. It is not an optimization system. It is the infrastructure that makes the other systems’ output safe to deploy. The core problem it solves: AI-generated kernels cannot move from a benchmark into production without a standardized path from generation through validation through deployment. FlashInfer-Bench provides that path.

The framework defines a unified schema called FlashInfer Trace, which captures kernel definitions, workloads, implementations, and evaluation results in a format that agents can generate and serving systems can consume. Kernels submitted to the framework pass through a correctness- and performance-aware benchmarking process built on real serving traces. A public leaderboard tracks which agent systems produce the best-performing kernels for specific inference workloads. An apply() mechanism provides dynamic substitution of optimized kernels into production serving engines including SGLang and vLLM, without requiring engine restarts or model redeployment.

The importance of FlashInfer-Bench becomes clear when reading WarpSpeed’s correctness findings. If a kernel passes a verifier but exploits narrow input distributions or encodes hardcoded constants, the apply() mechanism and its associated test suite provide a second line of defense before that kernel reaches a production request. The Atrex Kernel Agent operates on SOL-ExecBench and Atrex-Bench layouts; the Agentic Kernel Optimization work (arXiv:2608.14560) used the FlashInfer-Bench protocol for correctness-gated evaluation on NVIDIA B200 GPUs, achieving 92.68x speedup on Fused MoE, 1101.02x on DSA TopK Indexer, and 181.35x on DSA Sparse Attention, while consuming approximately 1.9 billion agent tokens. That token count deserves attention: autonomous kernel optimization at this depth is computationally expensive in its own right, and the cost does not always appear in the papers reporting the speedups.

How They Compare

What This Category Reveals

The entries above are ordered from most-to-least architectural scope, beginning with systems that optimize the full serving stack from kernel through feature store and ending with the deployment infrastructure that makes kernel substitution repeatable. That ordering reveals something: the most architecturally ambitious systems (FlashVector, Z.ai) are also the most production-deployed. The narrower systems (WarpSpeed, Atrex) show the strongest raw performance numbers on isolated kernel benchmarks. These are not contradictory findings. They reflect different definitions of the optimization target.

The critical ingredient across all of these systems is not a stronger code-generation model. It is a dense experimental feedback loop. Correctness tests, profilers, hardware counters, and end-to-end workload measurements become the agent’s environment. Without those signals, generation is guessing. With them, the agent can reject incorrect changes mechanically, retain only improvements, and continue for as many iterations as the compute budget allows. That feedback loop is what makes inference infrastructure unusually well suited to agentic optimization.

The category is splitting along two axes. On the scope axis: kernel generator, profile-guided kernel optimizer, whole-serving-stack optimizer. On the hardware axis: NVIDIA-only, AMD-specific, multi-hardware (NVIDIA, AMD, T-Head, MTIA). Systems that abstract across hardware targets require more complex profiler integration but offer a structural advantage for operators who want optimization to follow the workload rather than the hardware vendor. Meta’s KernelEvolve (arXiv:2512.23236), deployed in production across NVIDIA, AMD, and Meta’s MTIA accelerators, achieved 1.25-17x speedups across diverse workloads with kernel development time reduced from weeks to hours, and is worth tracking as the multi-vendor template matures.

Limitations and Open Questions

Several claims in this space require careful reading. Z.ai’s 3x throughput figure and two-week bring-up timeline are vendor-reported without independent verification. FlashVector’s results come from a single production deployment at Unity. WarpSpeed’s benchmark wins include the caveat that standard verifiers permit correctness exploits; the research team found eight kernels that passed verification but failed under fresh random seeds. The Agentic Kernel Optimization work, which achieved 92.68x, 1101.02x, and 181.35x speedups on three FlashInfer-Bench workloads, consumed approximately 1.9 billion agent tokens to do so. Autonomous kernel optimization at depth is expensive, and that cost does not appear consistently in the papers reporting speedup numbers.

Most speedups are workload-specific and hardware-specific. A kernel that achieves 14.9x on a Blackwell quantization workload may deliver 1.1x on a different attention pattern on a different GPU. The comparison table reflects this: no single system has the best result across all dimensions because no single system targets all dimensions. The hardware portability problem remains largely unsolved. CUDA expertise built up over a decade does not transfer to AMD HIP or T-Head PPU through model training alone; it requires deliberate integration of hardware-specific profiler tools and correctness harnesses.

The open question the category has not yet answered is economic. If running a kernel optimization agent costs 1.9 billion tokens, how does that cost compare to the value of the serving throughput improvement, amortized over the lifetime of the deployment? For large-scale production deployments like Z.ai’s 100,000-accelerator setup, the math may work clearly. For smaller deployments, the break-even point is less clear. The field currently reports speedup numbers more consistently than it reports optimization cost, which makes it difficult to evaluate these systems against the alternative of hiring a human kernel engineer.

What This Means for Engineering Teams

For teams building or operating LLM inference infrastructure, the immediate practical question is not “which of these systems should we deploy?” Most are not yet packaged as drop-in tools. The practical question is: what does this category tell us about where the next 2-3x of serving efficiency will come from? The answer from this comparison is that isolated kernel tuning is no longer the ceiling. FlashVector shows that model-server C++ and feature-processing Python are viable optimization targets; GEAK v4 shows that profiling at the serving level reveals bottlenecks invisible at the kernel level.

Teams evaluating these tools should weight five dimensions: optimization scope (kernel-only vs full stack), hardware target coverage, correctness gate rigor, whether the system reports results on end-to-end serving workloads or isolated microbenchmarks, and computational cost of the optimization run itself. A system that reports 1000x speedup on an isolated kernel but costs 2 billion tokens and runs only on NVIDIA hardware may be less useful than a system that achieves 2x serving throughput on the hardware the team already runs.

Understanding how LLM inference works at the serving layer is increasingly necessary as optimization agents begin to treat each layer as an independent target. Teams that have already evaluated model compression and quantization strategies are better positioned to judge whether agentic kernel optimization complements or substitutes for those approaches: compression reduces the computational load per request, while kernel optimization squeezes more performance out of each computation. The two often compound. If your team is working through the economics of AI serving at scale, the AI engineering services context makes these tradeoffs concrete.

Key Takeaways

• FlashVector (September 15, 2026) achieved up to 2x model-server throughput and 1.6x feature-store throughput at Unity by applying optimization agents to model-server C++ and Python feature-processing code, not just GPU kernels.
• Z.ai deployed a GLM-5.3-powered infrastructure agent across 100,000+ accelerators, reaching approximately 3x end-to-end throughput in under two weeks, with humans retaining control of objectives and production approval decisions.
• AMD GEAK v4 profiles running SGLang and vLLM workloads and improved serving throughput by 60%, 96%, and 42.2% on three configurations by targeting bottlenecks visible only at the serving level, not in isolated kernel benchmarks.
• WarpSpeed beat NVIDIA’s optimized baselines on 90% of 235 Blackwell kernels (2.24x geometric-mean speedup) in one day, and also found that standard benchmark verifiers permit seed overfitting and input distribution exploitation.
• One published kernel optimization study (arXiv:2608.14560) consumed approximately 1.9 billion agent tokens; compute cost of autonomous optimization remains an underreported dimension across this field.
• The category is splitting by scope (kernel vs full stack) and hardware breadth (NVIDIA-only vs multi-vendor); systems targeting the full serving stack and multiple hardware vendors are harder to build but address more of the actual performance gap.
• FlashInfer-Bench’s apply() mechanism makes correctness-gated kernel substitution into production engines like SGLang and vLLM practical, providing the deployment infrastructure that turns benchmark results into actionable serving improvements.

Work With Origins AI

Origins AI builds production AI infrastructure for engineering teams. If your team is evaluating inference optimization strategies or planning deployment across multiple hardware targets, talk to our team.