Google AX v0.3 Moves Agent Task State Out of Kubernetes etcd

On September 20, 2026, Google published AX v0.3.0, a substantial rewrite of its open-source agent orchestrator. The central argument is architectural: Kubernetes was not designed to be the control plane for millions of short-lived agent tasks, and etcd cannot absorb the resulting pressure at scale. AX v0.3 addresses this by removing agent task state from Kubernetes entirely, routing it through Redis instead, and delegating the execution layer to Agent Substrate, a separate runtime that treats each agent as a stateful logical actor rather than a container lifecycle.

Why Kubernetes Starts to Break Under Agent Workloads

Most infrastructure teams running AI agents today reach for Kubernetes first. It is familiar, well-supported, and provides all the primitives that work well for microservices: container scheduling, health checks, rolling deployments, autoscaling. For agents that run briefly, complete a discrete task, and exit, Kubernetes jobs or pods are a reasonable fit.

The problem appears when agent populations grow large and tasks become persistent. Kubernetes stores cluster state in etcd, and etcd has hard physical constraints. The AX design document states the issue directly: “Storing millions of short-lived tasks as Kubernetes CRDs pushes etcd past its comfort zone (single-digit GB storage limits, write-rate bottlenecks, control plane degradation).” Each task update, each status transition, each checkpoint becomes a write to etcd’s Raft log. At low volumes this is invisible. At millions of concurrent agents, it degrades the entire control plane, not just the agent workload.

There is a second structural mismatch. Containers are optimized for services that do work continuously. Agents frequently wait: waiting for a user reply, for a tool response, for a downstream API. A container that holds its position in memory while doing nothing still consumes compute. If most of a million agents are idle at any given moment, the pod-per-agent model wastes most of its allocated resources.

What AX v0.3 Changes: The Core Architecture

AX resolves both problems by separating concern between three distinct layers: the API server, the control plane, and the execution substrate. The DESIGN.md describes the flow:

  ax apply -f task.yaml
          │
          ▼
      ax-server
  (gRPC API + /healthz)
          │
   store & publish event
          │
          ▼
        Redis
  (Task Hashes + Event Streams + PubSub)
          │
    XREADGROUP (Streams)
          │
          ▼
      ax-controller
  (Horizontally Scaled Workers)
          │
    gRPC (Control API)
          │
          ▼
     Agent Substrate
  ┌──────────────────────────────┐
  │ • Atespace Provisioning      │
  │ • Actor Creation & Activation│
  │ • Worker Assignment          │
  │ • Egress Policy Filtering    │
  └──────────────────────────────┘

The ax-server is deliberately stateless. It validates incoming manifests, persists them to Redis, and publishes events. No task state lives in this process; it can be scaled or replaced without losing execution context. Redis holds the actual state: task hashes, event streams, and pub/sub channels.

The ax-controller pool consumes from Redis Streams using XREADGROUP, which gives each controller reliable message delivery with consumer group semantics. If one controller instance fails, its pending messages are redelivered to another. Scaling the controller pool is a replica count change, with no coordination required between instances. This is the work queue that replaces etcd as the reconciliation backbone.

The Four Declarative Primitives

AX introduces four API objects, each managed through a full CRUD plus watch interface over gRPC:

• Task — the unit of work. Represents one running agent with its current state, conditions, and execution history. Supports SuspendTask and ResumeTask as first-class RPCs.
• Workspace — the persistent file system context for a task. Survives suspension and reconnects on resume, so agents can pick up file state where they left off.
• Gateway — controls network egress. Rather than relying on cluster-wide network policies, each agent workspace has its own gateway that enforces which external destinations it may reach.
• Model — configuration for the AI model a task uses: endpoint, credentials, parameters. Stored as a named object in Redis rather than embedded in task definitions, so model configuration changes without redeploying agent code.

Making these objects independent rather than properties of a monolithic agent definition has practical consequences. A team can rotate model credentials by updating one Model object, audit all egress policies centrally through Gateway objects, and attach a fresh workspace to a resumed task without reconstructing any other configuration.

Agent Substrate: Virtual Memory for Agent Execution

Below the controller layer sits Agent Substrate, the runtime that does the actual compute multiplexing. The core idea is straightforward to state and consequential in practice: agents spend a large fraction of their time waiting. Substrate exploits this by maintaining a population of “actors” that far outnumbers the available “workers.”

When an actor becomes idle, Substrate checkpoints it: RAM state and local filesystem are snapshotted. The worker is freed for another actor. When work arrives for the suspended actor, a worker is assigned, state is restored, and execution continues. According to Google’s announcement on the Cloud Blog, Substrate claims 10× higher sandbox density compared to standard container runtimes, sub-500 ms restore latency, and more than 500 suspend/resume activations per second. These are vendor-reported figures without a published benchmark methodology, and the runtime is explicitly not production-ready at this stage.

Structurally, this is analogous to how operating systems page processes out of RAM when physical memory is under pressure. The kernel does not kill idle processes; it moves them to storage and restores them on demand. Substrate applies the same principle at the agent level: a logical agent is a persistent identity, but its physical compute allocation is temporary and shared.

From the atespace layer, Substrate handles provisioning, actor lifecycle, worker assignment, and egress policy enforcement. Atespaces appear to be namespacing units that group tasks and their associated resources, though this abstraction is still evolving.

What the Binaries Actually Are

AX ships four binaries. Understanding what each one does is useful because the decomposition reflects the architectural philosophy of the system:

• ax — the developer CLI. Applies task manifests with ax apply -f task.yaml, inspects and watches resources, and opens tunnel connections to the cluster.
• ax-server — a stateless gRPC service on port 8080. It validates manifests, persists state to Redis, and publishes events. Health endpoint: GET /healthz returns 200 OK.
• ax-controller — horizontally scaled reconciliation workers. Each instance reads from the Redis Stream, provisions atespaces and actors on Agent Substrate, applies egress policies, and drives tasks toward desired state. Add replicas to increase throughput.
• ax-task-runner — the entrypoint inside every task container. Bootstraps the workspace, serves metadata to the agent, and runs the agent command. Custom images can embed the underlying runner package directly instead of wrapping this binary.

Results and Availability

AX v0.3.0 is published under Apache-2.0. The repository includes CLI, control plane, Kubernetes deployment manifests, the Redis-backed state layer, Agent Substrate integration, and reproducible demos. The repository has 3.9k stars and 178 forks as of publication.

Agent Substrate is separately available on GKE in limited general availability for production access, with evaluation access more broadly. Google’s Cloud Blog positions the runtime as suitable for other Kubernetes infrastructure as well, not exclusive to GKE.

AX also documents a deferred execution tier in its release notes: non-latency-sensitive agent workloads can be queued for off-peak execution, analogous to batch computing modes in existing cloud infrastructure. This is relevant for cost optimization in agent pipelines where real-time response is not always required.

Limitations and Open Questions

AX explicitly warns of major breaking changes across versions. The project documentation states the system is not production-ready. All performance claims, including the 10× density figure, sub-500 ms restore, and 500+ activations per second, are Google’s own assertions. No independent benchmark with defined workloads, hardware configurations, or reproducible test methodology has been published.

The atespace concept is not yet fully documented in the DESIGN.md. The relationship between atespaces and Kubernetes namespaces is unclear, and it is not specified how workspace state is stored or how large filesystem snapshots are managed at scale.

Redis becomes a critical dependency. A Redis failure or data loss event would affect all task state in the control plane, making Redis availability and persistence configuration essential operational concerns. The design document does not address replication or failover strategies for the Redis layer.

The checkpoint-restore mechanism also raises questions that will only be answerable at production scale: what happens to in-flight tool calls when a task is suspended? How does the system handle agents that hold external sessions (browser state, authenticated API connections) across a suspend event?

What This Means for Engineering Teams

The most immediate implication is about where to store agent state. Teams running agents as Kubernetes jobs or CRD-backed resources who are experiencing etcd pressure now have a reference architecture for moving that state into a queue-backed store. Redis Streams with consumer groups is not a novel pattern, but applying it to agent orchestration is a meaningful step away from treating the API server as a database.

The suspend/resume API is more significant than it looks. SuspendTask and ResumeTask as explicit first-class operations mean agent lifecycle can be driven programmatically: paused on a rate limit, resumed when a quota refreshes, suspended when waiting for human approval, resumed when input arrives. This is qualitatively different from a pod restart, because state survives the transition.

For teams building agentic automation at scale, the separation of execution identity from physical compute solves a real operational problem. Today, long-running agents that wait frequently either waste warm compute or require custom hibernation logic. A runtime that handles this as a scheduling primitive removes significant application-level complexity.

The DevOps implications run deeper than they first appear. Engineers exploring the intersection of AI and cloud infrastructure, as described in the shift from DevOps to AIOps, will find AX v0.3 is a concrete instantiation of a trend: infrastructure primitives being redesigned around the behavioral profile of AI workloads rather than adapted from patterns designed for stateless services. The container model is not disappearing, but AX argues it should not be the leaf node in an agent execution hierarchy.

Key Takeaways

• AX v0.3 removes agent task state from Kubernetes etcd entirely, storing it in Redis Streams to eliminate etcd write-rate bottlenecks at scale.
• The system introduces four independent declarative objects, Task, Workspace, Gateway, and Model, each with full gRPC CRUD and watch interfaces, decoupling execution configuration from agent code.
• Agent Substrate implements checkpoint/suspend/resume semantics at the runtime level, claiming 10× sandbox density and sub-500 ms restore latency (vendor figures, not independently benchmarked).
• The architecture separates Kubernetes (managing worker infrastructure) from the agent control plane (managing millions of high-churn logical agent actors), a materially different topology from embedding an agent framework inside Kubernetes.
• The system is Apache-2.0, available now, and explicitly not production-ready; major breaking changes are expected across versions.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If your team is designing agent infrastructure that needs to scale beyond what Kubernetes CRDs can support, talk to our team.