AMD Hyperloom Autonomously Tunes Inference Stacks, Delivering 1.73x Median Throughput on AMD GPUs

AMD published a detailed technical report on September 21, 2026 for Hyperloom, an open-source multi-agent system that autonomously profiles inference workloads, modifies serving configurations and GPU kernels, benchmarks every change, and stores validated results for future sessions. Across 16 heterogeneous workloads on AMD Instinct GPUs, Hyperloom achieved gains from 1.35× to 7.31×, with a median of 1.73×, without any human involvement in the optimization loop. The system has been applied to over 14,000 models.

Why the Naive LLM-Driven Optimization Loop Fails

Optimizing a serving stack follows a predictable loop: profile the workload, identify what’s suboptimal, fix it, measure again. Automating this with an LLM seems obvious. In practice, AMD’s engineers found that long optimization runs break down in three specific ways, each traced to the same root cause: the model is left as the only thing tracking something the run depends on.

The first failure is drift. A model running for hundreds of turns, editing a serving framework’s source tree, killing and restarting inference server processes, eventually loses coherence. The authors describe this directly: “By the time the context window fills up with the run’s own transcript, the original goal has quietly turned into something else. The model may even start fabricating benchmark numbers, because nothing in its environment grounds it.” The second failure is amnesia: the same model, on the same GPU, under the same framework, re-discovers the same environment variables, re-reads the same source code, and hits the same dead ends it found last week, because nothing carries knowledge between sessions. The third failure is unsafe behavior: permission to edit a framework’s source tree, exercised across a long run, means a bad patch can land in the wrong file and corrupt every measurement thereafter.

The AMD blog states plainly: “None of these problems can be fixed with a better prompt.” Hyperloom was built as a multi-agent harness that addresses each failure structurally, not through prompting.

How Hyperloom Works

A Hyperloom session runs through five phases: Prelude, Framework Optimization, Kernel Optimization, Sweep, and Close. The Prelude phase measures a baseline on the stock configuration before touching the Recipe Knowledge Base, replays the closest previously validated recipe as a warm start, and builds a roofline analysis showing how far the workload sits from the GPU’s memory-bandwidth and compute ceilings. The roofline determines which directions are worth pursuing and how much headroom remains, feeding directly into later decisions about whether to open a second optimization cycle.

Framework Optimization has two jobs. The first is enablement: if the model fails to launch, Hyperloom classifies the failure and follows a six-step recovery ladder from least to most invasive — read-only diagnosis, serving-flag change, in-tree source patch, alternate wheel or source checkout, source localization of a merged upstream pull request, and finally a rebuild of compiled components like AITER (AMD’s library of optimized AI operators) or framework-specific kernel packages like sgl-kernel and vLLM. The second job is speed: Hyperloom searches the framework’s configuration space including serving flags, environment variables, weight and KV-cache precision, attention implementation, batching and scheduling limits, and open upstream pull requests ranked by expected throughput impact — applied as diffs, tested, and measured.

Kernel Optimization delegates to one of two AMD-specific backends per phase. GEAK autonomously selects and optimizes kernel targets across the entire phase. KernelForge applies specialized optimization passes — tuning, fusion, rewriting, or communication collectives — to targets Hyperloom selects from the profiling trace based on GPU time and rewrite feasibility. Both backends return proposed kernels; Hyperloom re-measures every accepted kernel change end-to-end under its own benchmark protocol. A speedup claimed by a backend is not accepted as ground truth: independent re-validation is mandatory before any kernel enters the session baseline.

Four Agent Roles for Goal Integrity

Four agent roles coordinate a session. The Orchestration agent runs as one continuous conversation for the entire session, maintaining the mission and progress summary in a state file rather than the transcript. At every turn, it reconstructs its state from that file rather than relying on accumulated context, preventing transcript noise from becoming the source of truth. The Critic runs at every keep-or-revert decision, fresh each time with no accumulated context, writing the record of what the run learned from that verdict. The Robustness agent runs when the session looks stalled, crashed, or circular — it acts as a circuit breaker that forces recovery instead of another loop iteration. The Specialist agent runs only where authoring judgment is needed, producing a reviewed diff, and is discarded immediately after. Its statelessness is intentional: with nothing accumulated, it has nothing to drift with.

No benchmark number reported by any agent is accepted as ground truth. The blog states: “Benchmarks are run by the coordinator, not requested by an agent. A gain the model predicts gets logged for calibration but never decides anything on its own, and any speedup a kernel backend claims is held as unverified until Hyperloom re-measures it end to end.”

The Recipe Knowledge Base

Every session reads from the Recipe KB before starting and writes back when done. Each row covers one workload: the winning configuration, achieved throughput, what the run learned, and what didn’t work — filed under seven fields: model, hardware, framework, model type, architecture, framework version, and precision. An exact match on all seven is a direct hit, replayed at full confidence. Most matches aren’t exact, so the lookup relaxes one field at a time down a defined fallback order. Warm replay contributed modest direct throughput gains in the detailed example session, but five of its six inherited flags remained in the final validated configuration, and the largest framework optimization required an attention backend that warm replay had already enabled.

Results

The paper documents one session end-to-end: Llama-3.1-8B-Instruct on a single MI355X, vLLM, bf16, 1024 tokens in and out, concurrency 64. Starting from a baseline of 7,677 tokens per second per GPU, the session proceeded stage by stage. Warm replay brought it to 7,765 (+1.15% cumulative). Framework optimization — adding AITER unified attention, FP8 KV cache and weights, and n-gram speculative decoding — raised it to 24,713 (+221.91% cumulative). Kernel optimization with one rewritten cache kernel brought it to 25,238 (+228.75% cumulative). A second optimization cycle, which added RoPE and KV-cache fusion plus a graph-capture cap, reached the final validated throughput of 26,682 tokens per second — a 3.48× gain from baseline, measured as a single number across the whole stack. The run closed with approximately 5 of its 24 hours unused because it had converged: it ran out of promising hypotheses to test, not time.

Isolated kernel speedups give a separate comparison: GEAK achieved a median 2.22× and mean 2.45× with a peak of 23.88×. KernelForge achieved a median 3.09× and mean 2.97× with a peak of 24.06×. Both outpaced a Claude Code baseline that achieved median 1.73× and peak 14.38× on isolated kernel optimization tasks.

The 16-workload fleet evaluation covers models from 752M (Qwen3-0.6B) to 862B parameters (DeepSeek-V4-Pro), spanning vLLM, SGLang, and xDiT, covering dense and MoE models, text and image generation. The highest gain was 631.5% (7.31×) on DeepSeek-V4-Flash-0731 (304B) under SGLang with bf16 at TP 4. The lowest was 35.2% (1.35×) on ERNIE-Image-Turbo (8B) under xDiT. The median across all 16 was 1.73×. The blog notes: “Tuning that used to take a scarce specialist weeks now finishes on its own, which turns pointing the harness at a new model into a scheduling decision instead of a staffing one.”

Limitations and Open Questions

All large-scale performance evidence currently comes from AMD. The system is built around AMD Instinct hardware, and the Recipe KB, kernel backends, and validated configurations are hardware-specific. There are no independently reproduced academic benchmarks for Hyperloom, and its performance on other hardware vendors’ GPUs is not evaluated.

Running a Hyperloom session consumes both GPU time for benchmarking and model-call costs for the orchestration and specialist agents. For teams on tight optimization budgets, the cost of running Hyperloom on a new model needs to be weighed against the expected throughput gains. The session described in the paper ran for up to 24 hours, which is a significant resource commitment for initial optimization of a given workload.

The Recipe KB improves as more sessions run, which means teams deploying the first session of a new model-GPU combination won’t benefit from accumulated experience — the value compounds over time, not immediately. Early sessions on novel model families start closer to vendor defaults.

What This Means for Engineering Teams

Model serving currently contains a substantial amount of hidden engineering labor. Every new architecture can require new kernels, new scheduler settings, new quantization choices, and framework patches that are discovered by specialists through manual trial and error. Hyperloom makes that loop machine-operated and — importantly — accumulates the results across sessions. A configuration validated for DeepSeek-V4 on MI355X becomes a warm starting point for the next DeepSeek variant rather than work that must be rediscovered from scratch.

For teams already managing the complexity of agent serving stacks, Hyperloom represents a shift in where optimization effort goes: from per-model manual tuning to building and maintaining the optimization harness itself. The Recipe KB means improvements compound across deployments rather than being one-off discoveries that live only in an engineer’s notes. Teams currently working on inference stack optimization for AMD hardware will find Hyperloom’s MIT-licensed source directly usable. The ROCm Docs documentation covers installation, configuration, and the full session workflow including GEAK and KernelForge setup.

Key Takeaways

• Hyperloom delivers a median 1.73× throughput gain across 16 unattended AMD Instinct workloads, with a range of 1.35× to 7.31× (7.31× on DeepSeek-V4-Flash-0731 under SGLang).
• One documented session — Llama-3.1-8B-Instruct on MI355X — went from 7,677 to 26,682 tokens per second through framework optimization, kernel rewriting, and a second optimization cycle.
• Four agent roles (Orchestration, Critic, Robustness, Specialist) separate concerns: only Orchestration runs continuously; the others are stateless and created only when needed.
• No model-reported benchmark number is accepted as ground truth: all performance claims are independently re-measured by the coordinator before entering the Recipe KB.
• Isolated kernel optimization: GEAK achieves median 2.22× and KernelForge achieves median 3.09×, both outperforming a Claude Code baseline at median 1.73×.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If your team is running AI inference at scale and needs to move past manual per-model tuning cycles, talk to our team.