Six Systems Blurring the Line Between Agent Orchestrator and Inference Engine

On September 16, 2026, Yipeng Liu, Yingqiang Zhang, Feifei Li, and Huanchen Zhang submitted a paper with a pointed argument: inference engines are making guesses about tool-call durations that no pre-call estimate can make accurately, while the running tool already holds the answer. Their system, Ask the Tool, Don’t Guess, routes live progress signals from executing tools directly into the KV-cache retention logic of the serving engine, cutting p90 time-to-first-token after a tool call by 20.7% against LRU, close to an oracle. That result is notable not because the number is large, but because of where the information comes from: outside the model server, inside a running tool. A boundary that once seemed fixed has been crossed.

The six systems examined here each cross it differently. Some redesign the interface between orchestrator and engine; some give the agent policy direct commands over GPU memory; one caches tool schemas so they become reusable across any call combination; another introduces TTL semantics so KV state can survive tool waits without monopolizing GPU memory. What they share is a rejection of the premise that an LLM server is a stateless token-generation service and that an agent orchestrator is an external caller. The shared alternative: treat the agent program and the inference runtime as a single system that should be optimized together.

Why This Comparison Is Timely

The observation that agentic workloads strain conventional LLM serving has been around for two years. What changed between early 2026 and this week is that the proposed solutions are no longer isolated heuristics. They can now be grouped into three distinct approaches, each with independent benchmark evidence.

The first approach is architectural: change the interface between orchestrator and engine so the engine knows what the agent is doing. Sutradhara (January 2026, Microsoft) and ThunderAgent (February 2026) both went here. The second approach is mutable cache semantics: give the agent policy primitive commands to rewrite or expire GPU memory. Leyline (May 2026) and Continuum (November 2025, Berkeley) represent this branch. The third approach is composition-invariant encoding: pre-compute tool schema KV blocks so they survive being called in any order or combination. ReCache (August 2026) takes this path. Ask the Tool is the newest arrival and fits a fourth slot: real-time information transfer from the executing tool to the serving engine.

Together, these six papers constitute a systems agenda, not a collection of independent optimizations. Understanding how LLM inference engines handle memory and scheduling under standard workloads is the starting point; these systems then explain what breaks when that workload becomes agentic.

The Shift: Stateless Token Server to Program-Aware Runtime

A conventional LLM serving engine treats every request as independent. It receives a prompt, generates tokens, manages KV memory using policies tuned for single-turn or chatbot traffic, primarily append-only caches with LRU eviction, and returns a response. The orchestrator above it decides which prompts to send, in which order, and what to do with tool outputs.

Agentic workloads break three assumptions this design relies on. First, tool calls create pauses of hundreds of milliseconds to minutes during which the engine holds KV state that cannot yet be used. Evicting that state to free GPU memory means expensive re-prefill when the tool returns; retaining it wastes capacity. Neither choice is obviously right without knowing how long the tool will run. Second, context reuse across agent turns does not follow prefix-sharing patterns. The agent edits its context between iterations: it drops failed tool outputs, retries calls, adjusts its trajectory. Exact-prefix caching misses reuse that is semantically real but positionally invalidated. Third, tool schemas, the structured definitions of what tools accept and return, appear in almost every agent call, in varying combinations and orders, making prefix caching ineffective for what is often the longest constant section of the prompt.

The six systems each address one or more of these gaps. The interesting question is not which one performs best in isolation, but what they reveal together about how much information the inference runtime should receive from the program invoking it.

ThunderAgent: LLM Programs as the Unit of Scheduling

ThunderAgent, submitted February 14, 2026 by Hao Kang, Ziyang Li, Weili Xu, and colleagues at Georgia Tech, starts from the observation that current systems schedule LLM calls and tool calls on a per-request basis, treating KV caches and tool execution environments as separate concerns managed by separate subsystems. The orchestrator (Kubernetes, Ray, a custom agent loop) is unaware of what the engine is doing with GPU memory; the engine is unaware of which tools the orchestrator is about to call or what state those tools need.

ThunderAgent introduces the concept of an LLM Program: a unified representation of the entire agent workflow that encompasses KV caches, system states (the agent’s internal variables and context), and external tool assets, specifically disk memory and network ports. By making all of these resources visible to a single scheduler, ThunderAgent can optimize across what were previously isolated decisions.

Two components do most of the work. The program-aware scheduler uses workflow-level knowledge to maximize KV cache hit rates and prevent memory imbalances that arise when some requests hold large KV states while others queue waiting for GPU memory. The tool resource manager handles asynchronous preparation of execution environments: rather than waiting for one tool call to complete before setting up the next, it prepares tool environments in advance based on predicted execution paths. The authors report 1.5–3.6× throughput improvement in serving across coding, routing, and scientific discovery agents, 1.8–3.9× improvement in RL rollout settings, and up to 4.2× disk memory savings over comparable systems. The code is available at github.com/Agentic-Kinetics/ThunderAgent.

What distinguishes ThunderAgent from the other systems here is scope. It is not an optimization to one component of the serving stack; it is a redesign of the resource boundary between agent program and serving infrastructure. The LLM Program abstraction is the key move, it creates a shared representation that the scheduler can act on rather than inferring from incomplete local state.

Sutradhara: Three Optimizations Through a Thin Co-design API

Sutradhara, submitted January 19, 2026 by Anish Biswas, Kanishk Goel, Srivarshinee S, Jayashree Mohan, and colleagues at Microsoft, takes a different path: instead of redesigning resource ownership, it adds a thin API between an existing orchestrator and an existing inference engine (vLLM) and uses that API to enable three specific optimizations.

The authors start from production trace analysis. Their data shows that tool calls account for 30–85% of First Token Rendered (FTR) latency in agentic workloads. KV cache hit rates collapse despite context reuse across iterations, because the orchestrator and engine cannot coordinate on what to keep. And sequential orchestration, waiting for a complete tool output before sending the next LLM call, wastes time that could be used for prefill.

The first optimization, tool-aware prompt splitting, allows the engine to begin prefilling the next LLM call while the current tool is still running. The orchestrator signals the engine that a tool call has been dispatched; the engine splits the in-progress prompt and starts processing the parts it already has. The second optimization, streaming tool execution, dispatches tools incrementally during decode rather than waiting for a complete tool output. This reduces the wall-clock gap between when the model finishes generating a tool call and when execution begins. The third optimization, orchestrator-aware cache management, passes semantic hints from the orchestrator to the engine so the engine can make better eviction decisions. Rather than guessing based on access patterns alone, the engine receives explicit information about which cache entries the orchestrator expects to reuse.

On A100 GPUs, Sutradhara sustains up to 77% higher load at the same median FTR latency, or reduces median FTR latency by up to 15% at the same load. End-to-end latency improves by up to 11%. These numbers come from production traces, which is a meaningful difference from synthetic benchmarks, the workload distribution reflects real agentic usage patterns. The system is implemented on vLLM, making it compatible with an existing production inference stack without a ground-up rewrite.

Continuum: TTL Semantics for KV State During Tool Waits

Continuum, originally submitted November 4, 2025 by Hanchen Li, Runyuan He, and colleagues at UC Berkeley (Ion Stoica’s group, with contributions from Huanchen Zhang, Alvin Cheung, and Joseph Gonzalez), and revised most recently on September 8, 2026, attacks the most direct consequence of agent tool waits: the engine evicts KV cache that could be reused moments later when the tool returns.

The core mechanism is a time-to-live value attached to KV state belonging to sessions waiting on tool calls. When an agent dispatches a tool, Continuum calculates a TTL based on two factors: the estimated cost of recomputing or reloading that KV state if it is evicted, and the expected queueing delay that would result from eviction. If retaining the KV state is cheaper than the combined cost of eviction and re-entry, the TTL is set accordingly and the state is pinned in GPU memory. When the TTL expires, the cache is automatically evicted, providing a safety valve against edge cases where a tool runs far longer than expected.

The combination with program-level first-come-first-serve scheduling preserves multi-turn continuity: sessions that are waiting on tools maintain their position in the queue and resume without being treated as new requests. Evaluated on real-world agents, SWE-Bench, BFCL, and OpenHand, with Llama-3.1 8B and 70B, Gemma-3 12B, and GLM-4.5 355B, Continuum improves average job completion time by over 8× while also improving throughput. The wide range of model sizes and benchmarks tested is worth noting: most papers in this space evaluate on one or two settings.

The limitation Continuum acknowledges is also the most honest one in this category: tool-call duration is inherently unpredictable, and TTL values are estimates. The system is designed to degrade gracefully, the automatic expiry prevents indefinite GPU memory retention, but the TTL calculation requires workload-specific calibration.

Leyline: Declarative Cache Edit Directives

Leyline, submitted May 31, 2026 by Bole Ma, Jan Eitzinger, and Harald Koestler, takes a different angle on the mutable-cache problem. Its starting observation is that agents do not just add to their context across turns; they actively edit it. Failed tool calls get retried. Stale outputs get dropped. The agent’s trajectory pivots. Under a standard append-only cache model, each edit forces a complete re-prefill of everything that came after the edited position, expensive even for short contexts, prohibitive for long ones.

Leyline introduces a serving-side primitive: a declarative directive 4-tuple that separates what to edit from how to preserve attention correctness afterward. The agent policy declares the edit (a span of cached content to remove or replace) and a mode: either an in-place splice, which retains subsequent KV state and applies a closed-form RoPE rotation correction to restore attention math, or a prefix-trimmed re-prefill, which semantically forgets the removed span. An architecture-agnostic interface routes the directive to a per-architecture kernel that applies the correction.

The splice kernel achieves a +11.2 percentage-point improvement in replay cache-hit rate and reduces latency by up to 241 milliseconds compared to the re-prefill baseline. A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 percentage points on debug-gym, a benchmark of iterative debugging tasks where the agent’s ability to selectively forget prior context is central to performance. The mechanism’s value is not the truncation rule itself, it is the primitive that makes any such rule expressible without touching the serving engine’s internals.

What Leyline provides that the other systems here do not is genuine mutable cache semantics: the agent policy can issue commands that change what is stored in GPU memory, not just influence eviction timing. The paper explicitly frames the policy space this primitive enables as “the agenda”, meaning the authors view the kernel and interface as infrastructure, not a finished system.

ReCache: Composition-Invariant Tool Schema Caching

ReCache, submitted August 20, 2026 by Yichu Fang, Sitong Wei, Haozhe Hu, and Xiaoyu Shen, targets a different inefficiency: the tool and skill schemas that appear in nearly every agent prompt are encoded fresh on each call, even though their content is constant. Standard prefix caching cannot reuse their KV states when schemas appear in different combinations or orders across requests, the prefix match fails the moment the order changes.

The solution is resource-wise attention, a modification to the attention mechanism that removes cross-resource interactions between tool schemas and assigns resource-local positions to each schema block. The result is KV blocks that are composition-invariant: a schema’s KV state is the same regardless of which other schemas appear in the same prompt or in what order. These blocks can be cached once and reused across any call configuration.

ReCache then applies two pruning strategies on top of this. Structural pruning restricts each resource’s visibility to a selected subset of layer–KV-head-group routes based on its contribution to the final prediction. Semantic pruning retains only invocation-critical fields from the schema definition. Together, these reduce both the memory footprint and the attention computation required for each schema block.

The measured results are striking. Resource-wise attention matches dense invocation performance (82.3% Inv-F1 versus 82.4% for full dense attention), while delivering a 3.655× TTFT speedup. The complete framework reduces allocated KV-tensor memory by 92.43% and accelerates attention by 1.423×. The benchmark covers seven public tool-use and skill-use datasets with resource-disjoint test splits, meaning the schemas seen at test time were not present during training. Code is available at github.com/EIT-NLP/ReCache.

The interesting caveat is architectural: resource-wise attention requires modifying the attention computation, which means it cannot be dropped into an existing engine as a scheduling policy. It requires model-level changes, or at minimum careful integration with the engine’s attention kernel.

Ask the Tool, Don’t Guess: Real-Time Progress as a Serving Signal

The September 16 paper from Yipeng Liu, Yingqiang Zhang, Feifei Li, and Huanchen Zhang is the most recent entry and the one that sharpens the framing of this entire category. Its central claim is empirical rather than architectural: no estimate that is fixed before a tool call starts can accurately predict its duration. Duration depends on runtime conditions, network state, data size, service load, that are unknowable before execution begins. Published predictors based on tool name, call history, or declared-before-start durations are, the paper shows, not just imprecise but unreliable in ordering calls by expected duration, which is what serving engines actually need.

The proposed solution is to have the running tool report its progress while it runs. A census of four public agent corpora finds a readable progress signal in most tool execution time once it is exposed. The signal comes in two strengths: a fraction of remaining work (useful for gradual KV cache decisions) or a near-end signal (useful for initiating prefill in advance). A harness recovers this signal without changing what the agent framework sees, the agent’s benchmark score is unaffected, at no measurable cost.

Plugged into a production engine through small hints at KV-cache decision points, the live progress signal cuts p90 TTFT after a tool call by 20.7% (HBM-only configuration) and 20.8% (HBM + DRAM) against LRU, approaching oracle performance. The paper is careful about what this proves: the gain is from signal quality, not system redesign. The same cache decision logic performs near-optimally when given accurate information about what the tool is doing. The architectural implication follows: if the information exists inside the running tool, the serving system should have a channel to receive it.

How They Compare

What This Category Reveals

The entries are ordered above from broadest to narrowest architectural scope, which is not the same as most-to-least impact. ThunderAgent redesigns resource ownership across the entire serving stack; Ask the Tool adds a single signal channel between a tool and a cache manager. The breadth difference reflects a genuine trade-off: broader interventions can capture more optimization opportunities but require deeper integration; narrower ones can be adopted without changing the underlying engine.

What the six systems reveal together is that the serving engine’s information horizon is the binding constraint. Every gain reported here comes from giving the engine information it previously lacked: workflow structure, semantic cache hints, tool dispatch timing, TTL values, edit directives, or live progress signals. In each case, the engine’s existing logic, scheduling, eviction, prefill, performs significantly better when that information is available. The engine is not broken; it is operating on incomplete data.

The open question the category has not yet answered is standardization. Each system exposes a different piece of application semantics through a different interface. There is no common protocol for what an agent program should tell its serving engine, in what format, at what point in the execution. Sutradhara’s semantic hints, ThunderAgent’s LLM Program abstraction, Leyline’s directive 4-tuple, and Ask the Tool’s progress channel are not composable. A production system cannot easily adopt all four simultaneously. Whether these interfaces converge into a standard runtime contract or remain engine-specific optimizations is the central unresolved question in this space.

Limitations and Open Questions

Every system in this comparison reports gains that depend on workload characteristics their papers control. ThunderAgent’s 1.5–3.6× throughput range spans its coding, routing, and scientific discovery agent benchmarks, which differ substantially in tool-call frequency, context length, and parallelism structure. Extrapolating to a different workload distribution requires re-evaluation. Sutradhara’s numbers come from production traces at Microsoft scale; the degree to which those traces represent other organizations’ agentic workloads is unknown.

Continuum’s 8× job completion improvement is the most impressive number in this set. It should also be read carefully: it reflects the cost of the status quo (evicting KV state that is needed again minutes later) against a system that retains it with TTL bounds. The denominator matters, workloads with very long tool waits relative to GPU memory pressure show the largest gains. Workloads where tool waits are short or where GPU memory is underloaded would show smaller improvements.

Leyline is explicit that it provides infrastructure, not policy. The +14.3 pp solve rate gain on debug-gym comes from a ten-line truncation rule; whether that rule transfers to other agentic tasks, or whether better rules can be designed, is unanswered. ReCache’s 92.43% KV memory reduction is for the schema encoding specifically, not the full agent context. The full-context memory savings depend on how large the tool schemas are relative to the rest of the prompt.

Ask the Tool’s result is perhaps the most conditional of the six. It requires tool developers to instrument their tools to report progress. That is not a heavy technical burden, the paper demonstrates it adds no measurable overhead to agent benchmark scores, but it is an adoption barrier. Tools that call external services, run opaque binaries, or execute code written by end users cannot easily be instrumented. The census of four agent corpora finds readable signals in “most tool time,” which implies a meaningful fraction where no signal is available.

What This Means for Engineering Teams

If your team is running LLM inference workloads that are shifting from single-turn or chatbot patterns to multi-step agentic loops, these systems identify five concrete engineering decisions that standard serving infrastructure does not currently make well: when to evict KV state during tool waits, how to share cache hints from the orchestrator to the engine, how to handle agent context edits without full re-prefill, how to reuse tool schema encodings across call combinations, and how to time prefill relative to tool execution.

Of the six approaches, Continuum and Sutradhara are the most immediately applicable because they operate above the model-weight level: they require changes to the serving scheduler and orchestrator interface, not to the attention mechanism itself. ReCache requires attention kernel changes, which means it is more likely to be adopted through model providers or serving-framework integrations than by individual deployment teams. ThunderAgent’s LLM Program abstraction is the most powerful but also the deepest integration, it requires treating the agent program and serving infrastructure as a single system from the start.

Ask the Tool is worth tracking specifically because its gains come from signal quality, not system redesign. If tool instrumentation becomes standard practice in agentic frameworks, something framework maintainers could add without touching the serving engine, it provides near-oracle KV retention decisions with minimal integration cost. Building AI infrastructure that handles these tradeoffs well is the practical agenda; the systems engineering required to get there is more specific than generic “agent infrastructure” work, and the comparison dimensions above are a reasonable starting checklist for evaluating any proposed solution.

Teams building on vLLM should also track how inference infrastructure has evolved at production AI labs, the lessons from deployment at scale provide context for which optimizations matter most under real load versus synthetic benchmarks.

Key Takeaways

• Tool calls account for 30–85% of agentic FTR latency at production scale (Sutradhara, Microsoft traces), making the serving engine’s behavior during tool waits the dominant latency source.
• Continuum reports 8× lower average job completion time by pinning KV state during tool waits with TTL-bounded retention, evaluated on Llama-3.1 8B/70B, Gemma-3 12B, and GLM-4.5 355B across SWE-Bench, BFCL, and OpenHand.
• ThunderAgent achieves 1.5–3.6× serving throughput and 1.8–3.9× RL rollout improvement by treating the agent workflow as an LLM Program whose KV, system state, and tool assets are scheduled as a unified resource pool.
• ReCache cuts KV-tensor memory for tool schema encoding by 92.43% and delivers a 3.655× TTFT speedup by making schema KV blocks composition-invariant, with no meaningful accuracy loss (82.3% vs 82.4% Inv-F1).
• The six systems expose fundamentally different pieces of agent semantics to the serving engine; there is no common interface or standard today, and gains from one system cannot trivially be combined with gains from another.
• Ask the Tool demonstrates that the best KV retention decisions require runtime information from executing tools, pre-call estimates are provably insufficient, which suggests tool instrumentation, not just serving-engine tuning, is part of the infrastructure problem.

Work With Origins AI

Origins AI builds production AI infrastructure for engineering teams. If your agentic workloads are hitting latency walls that standard LLM serving tuning cannot solve, talk to our team.