Six Systems Turning Agent Memory From a Database Into a Learned Policy

On September 15, 2026, researchers at Harbin Institute of Technology posted a system where two agent networks co-evolve through online reinforcement learning to decide what to store and when to retrieve, not as fixed logic, but as a trained policy that improves with every conversation. That paper, ICML (Interactive Memory Learning), is the latest in a cluster of 2025-2026 systems converging on the same structural idea: agent memory should not be a passive archive with retrieval rules bolted on, but a separate learned subsystem with its own optimization objective. The difference matters because it shifts the question from “how do we retrieve better?” to “what should the agent have remembered in the first place?”

Why This Comparison Is Timely

The standard production pattern for agent memory is still store, embed, retrieve, and stuff into context. That architecture works when the relevant information is obvious at write time and when the answer model can sort through noisy retrieved context without penalty. Neither condition holds reliably in long-horizon tasks. As tasks extend across sessions, the store-everything approach fills retrieval with outdated or contradicted facts; the stuff-it-all-in approach taxes the answer model with irrelevant context; and neither approach learns from its own mistakes.

What changed in 2025 and 2026 is that several independent groups stopped treating memory as an infrastructure problem and started treating it as a policy learning problem. Memory-R1 (August 2025) trained an explicit CRUD operation manager with PPO and GRPO. UI-Mem (February 2026) coupled hierarchical experience memory with online RL for GUI agents. ATMem (June 2026) introduced a memory-cost-aware RL objective that teaches an agent when memory use is worth its overhead. MemChain (July 2026) trained a post-retrieval policy that transforms raw retrieved candidates into compact, grounded evidence. The ICML paper from September 2026 adds online adaptation: the memory policy itself keeps improving as interactions accumulate. Taken together, these systems define a category large enough to compare seriously.

The Shift: Passive Archive to Learned Controller

In the old model, memory is a service: the agent writes facts to a database, retrieves semantically similar ones at query time, and hands the results to the language model. The database does not know which facts will matter later. The retriever does not know whether the retrieved context will actually help. The language model absorbs whatever arrives.

In the emerging model, those three decisions, what to write, whether to retrieve, and how to transform retrieved content before passing it forward, are each treated as a policy with a trainable objective tied to downstream task performance. Some systems train all three; others focus on one. What they share is the premise that fixed heuristics will systematically fail as task horizons grow, and that reinforcement learning, with delayed or sparse rewards propagated back to earlier decisions, is the right tool for the job. The ReAct-style agent pattern that most developers use today treats memory as a retrieval call inside a reasoning loop; these six systems treat memory as a parallel learnable process that operates on its own reward signal and updates its own parameters.

ATMem: Active Task-Driving Execution State

Liu et al. (arXiv:2606.31612, June 30, 2026) make a pointed observation about mobile GUI agents: 83% of tasks on AndroidWorld involve data operations, yet existing memory systems store records of past observations rather than tracking the evolving status of task-relevant data items. If an agent retrieves the correct contact name but does not know whether that contact has already been processed, it will either repeat the operation or skip it entirely. The information is present; what is missing is its current role in the workflow.

ATMem replaces passive records with an Active Task-Driving Memory: a structured execution state that organizes task-relevant values under four fields. Workflow Progress tracks which app-level data files have been completed and which remain. Constraints encodes the instruction-defined conditions independently of the data schema. Schema defines the minimal data units required by the task, such as fields and their properties. ItemContent stores the observed content and execution status of individual data instances. The agent reads and updates this structure after every observation and operation, rather than querying a static store at retrieval time. ATMem is not a hand-engineered schema; it is induced from the task description and maintained by the agent throughout execution.

The training procedure has two stages. Supervised fine-tuning on verified trajectories teaches the agent to construct and update valid ATMem states and to reference them when predicting actions. STR-GRPO then teaches the agent when to use ATMem at all. It does this by running paired rollouts of the same task, one with ATMem enabled, one with it disabled, and estimating ATMem’s marginal contribution from the difference in task outcomes. A memory-cost-aware reward simultaneously penalizes invocations that add steps without improving the result. The agent learns to use structured state tracking selectively rather than by default. ATMem-UI-8B achieves 76.6% success rate on AndroidWorld, outperforming UI-TARS-2-230B (73.3%) while using roughly 1/29 of its parameters. On MobileWorld, the 4B variant already exceeds all non-ATMem baselines including 72B-scale reference models, with 20.5% versus the next best at 17.1%.

UI-Mem: Self-Evolving Experience Memory for Online RL

Xiao et al. (arXiv:2602.05832, February 5, 2026) from the Chinese University of Hong Kong and vivo AI Lab start from a different bottleneck. Standard online reinforcement learning for GUI agents fails because reward signals are sparse, credit assignment is hard across long trajectories, and the agent must rediscover the same failure modes from scratch on every new task. The solution is not a better reward function in isolation; it is a memory that accumulates structured experience and transfers it across tasks, so that the agent starts each new task closer to competence than it would from a blank slate.

UI-Mem maintains a Hierarchical Experience Memory with three levels: high-level workflows for planning, subtask skills for execution, and failure patterns for error prevention. These are stored as parameterized templates rather than raw trajectory records. When the agent encounters a new task, it retrieves relevant abstract templates and instantiates them with task-specific details. The template “Send email to {{recipient}}” becomes concrete once the current recipient is known. Hierarchical decomposition means the agent can receive credit for completing intermediate subtasks even when the full task fails, addressing the credit-assignment problem that causes standard GRPO to stall in long-horizon settings. The training dataset contains only 256 task instructions, a small base that the self-evolving loop extends over time.

Integrating this memory into online RL introduces its own hazard: if all rollouts in a GRPO batch receive full guidance and succeed uniformly, the advantage variance collapses to zero and the gradient signal vanishes. UI-Mem addresses this through Stratified Group Sampling, which constructs each training batch with trajectories receiving strong, weak, and no memory guidance. Some trajectories succeed with full plans; others must explore independently. This maintains the within-group diversity that GRPO requires for stable advantage estimation. A Self-Evolving Loop then extracts successful plans and failure diagnoses from new trajectories and updates the memory pool continuously. UI-Mem-8B with memory retrieval at inference time achieves 71.1% success rate on AndroidWorld, surpassing Gemini-2.5-Pro (69.7%) and Seed1.8 (70.7%). The 4B variant with inference-time memory reaches 62.5%, beating UI-Venus-7B (49.1%) at nearly half the parameter count.

Memory-R1: Structured CRUD Operations Trained With RL

Yan et al. (arXiv:2508.19828, August 27, 2025) from LMU Munich, TU Munich, Cambridge, and the University of Hong Kong identify the core failure mode of heuristic memory systems with a concrete example. When a user says “I adopted a dog named Buddy” in one session and “I adopted another dog named Scout” in a later session, a vanilla LLM memory manager misinterprets the second statement as a contradiction and issues DELETE + ADD, fragmenting the record into two separate, conflicting entries. A trained manager recognizes accumulation and issues a single UPDATE, consolidating the record as “Andrew adopted two dogs, Buddy and Scout.” The difference is not a bigger model; it is the right training signal.

Memory-R1 trains two specialized agents with PPO and GRPO. The Memory Manager learns to select among four explicit operations: ADD, UPDATE, DELETE, and NOOP, applied to a persistent external memory bank after each interaction. The Answer Agent applies a Memory Distillation policy: from 60 memories retrieved via RAG, it filters to the subset that actually supports the current question before reasoning. This separation keeps the answer model focused and prevents it from being distracted by memories that are technically relevant but not useful for the specific question. Both agents are trained on outcome-based rewards tied to final answer correctness. The training requires as few as 152 question-answer pairs, a data efficiency that matters for teams adapting the system to narrow domains without large annotation budgets.

On LOCOMO using LLaMA-3.1-8B-Instruct, Memory-R1-GRPO improves overall F1 by 68.9%, BLEU-1 by 48.3%, and LLM-as-a-Judge by 37.1% over Mem0. The gains hold on a different backbone: with Qwen-2.5-7B-Instruct, GRPO improves F1 by 57.3%, BLEU-1 by 41.5%, and LLM-Judge by 33.8%. Consistent improvements across two distinct LLM architectures suggest the RL training instills generalizable memory behavior rather than memorizing patterns specific to one model family. The temporal reasoning category shows the largest absolute gains, where the model must reconcile facts that changed across sessions, precisely the case that DELETE + ADD operations handle incorrectly.

MemChain: A Trained Post-Retrieval Policy

Ma et al. (arXiv:2607.24097, July 27, 2026) from the Institute of Automation, Chinese Academy of Sciences and Memorax AI target a specific gap: what happens after retrieval. Existing systems invest effort in what to write and which records to retrieve, but then hand the raw retrieved candidate set directly to the answer model. That model must resolve redundancy, staleness, and conflicting claims across records while simultaneously generating a response. The problem is not that retrieval fails to surface relevant records; it is that retrieved candidates are not yet in the form the answer model needs. MemChain moves this transformation step outside the answer model and trains a dedicated post-retrieval policy to perform it explicitly.

Given a question and a retrieved candidate set, MemChain runs three steps. First, it generates an evidence plan: a question-conditioned specification of which entities, temporal scope, and cross-record relations the answer requires. Second, it constructs a grounded evidence trace that organizes retrieved memories according to their semantic roles and dependencies, with each evidence statement linked to the source candidate IDs. Third, it applies explicit memory actions, including update, contrast, and resolve, to produce compact answer-facing active memory. Only this compact representation is passed to the frozen answer model; the candidate set never touches the answer model directly. MemChain is trained in two stages: supervised trace learning establishes structurally valid plans and traces, and TMPO (Trace-Guided Memory Policy Optimization) then optimizes for downstream answer quality while jointly rewarding trace grounding, structural validity, and answer stability across multiple rollouts.

The numbers make the efficiency argument clearly. MemChain passes 143.3 answer-facing evidence tokens on average versus 3,491.0 tokens for SimpleMem, a 24x reduction, while achieving higher accuracy. Memory-side latency is 0.83 seconds per question, including retrieval and post-retrieval evidence composition. On LoCoMo with a frozen Qwen3-14B answer model, MemChain (SFT + TMPO) reaches 80.26% overall accuracy, exceeding the strongest retrieval-based baseline by 19.09 percentage points. An ablation confirms that removing the grounded trace field alone costs 13.96 accuracy points, the largest single-component drop in the study. The policy backbone transfers: tested across Qwen3 sizes from 1.7B to 14B, TMPO improves over SFT consistently. Code is available at github.com/mayiwen0212/MemChain.

MMPO: Belief Entropy as an Intermediate Memory Reward

Liu et al. (arXiv:2605.30159, May 28, 2026) from USTC and Tencent diagnose a problem with how outcome-based RL trains recursive summarization policies. When a final task reward propagates back through a long sequence of summary decisions, it cannot identify which intermediate summary lost track of a critical fact or introduced a hallucination. The signal is too sparse. As context windows extend into hundreds of thousands of tokens, ambiguous intermediate summaries compound silently until the agent’s estimate of the task state diverges far enough from reality to cause a failure.

MMPO introduces Belief Entropy, a self-supervised proxy signal for intermediate summary quality, grounded in POMDP theory. In a POMDP, the agent maintains a belief over the latent task state. A high-quality summary should induce a confident, stable belief; a noisy or ambiguous summary should induce uncertainty. To measure belief uncertainty without access to ground-truth latent states, MMPO uses a metacognitive probe: it poses a task-state anchor question to the model and measures the entropy of the response distribution. The specific probe that works best asks jointly about task progress and missing information, which the authors call “progress + gap.” MMPO adds this Belief Entropy signal as a dense reward at each intermediate memory state, complementing the sparse final outcome reward, and penalizes summaries that induce high epistemic uncertainty.

On RULER-HotpotQA, MMPO improves over RL-MemAgent by an average of +3.14% for Qwen2.5-7B and +3.12% for Qwen2.5-14B across context lengths from 224K to 3.5M tokens. The gains concentrate at the longest contexts: +5.47% at 896K for the 7B model and +5.38% at 3.5M for the 14B. The system maintains 97.1% of peak performance at 1.75M-token contexts. On WebShop, MMPO reaches a reward of 77.25 versus MEM1’s 70.87, showing the benefit of dense belief supervision transfers from retrieval-based QA to interactive environment tasks. The interesting comparison is between the Progress + gap probe (82.98% accuracy) and the direct-answer probe (78.17%): rewarding low answer entropy too early encourages premature confidence before the agent has gathered sufficient evidence, making anchor question design a real engineering decision.

ICML: Online Memory Learning That Evolves With Each Session

Ke et al. (arXiv:2609.17088, September 15, 2026) from Harbin Institute of Technology and Pengcheng Laboratory make the most direct argument for memory management as a trained interactive process. All five other systems in this comparison either train offline on collected data or run RL episodes without updating the memory policy between real interactions. ICML trains the memory policy continuously during real conversations: the more sessions the agent completes, the better it becomes at selecting what to memorize and when to trigger retrieval, without any human re-labeling or batch retraining.

ICML frames long-term conversation memory as a Partially Observable Markov Decision Process and trains two Actor-Critic agents jointly. The Planner agent decides which information to encode after each session, selecting high-value memories over transient noise. The Trigger agent decides when to retrieve memory given the current utterance. The training signal is non-trivial because the Planner’s storage decisions matter only when the Trigger later uses a stored memory to improve a response, possibly many sessions later. ICML propagates this delayed feedback backward using a cross-session truth reward: the Planner’s policy is updated only when the Trigger successfully applies a stored memory to satisfy user expectations. This ensures the Planner learns to encode based on eventual utility rather than immediate saliency, which are often different.

A cold-start problem exists because a new agent has no memories when it first encounters a user and therefore produces suboptimal responses from the start. ICML addresses this through a retrospective session synthesis pipeline: starting from a seed interaction, the system works backward to generate consistent prior sessions, then forward-annotates them with high-quality expert memory decisions. This provides the initial expert data needed for rapid test-time adaptation in unseen scenarios. Evaluated on three long-term open-domain conversation datasets drawn from real human interactions, ICML outperforms static memory baselines across response quality and personalization measures, and it improves consistently as more interactions accumulate. The open question is stability: whether the online adaptation drifts toward recent preferences over very long horizons at the cost of older, still-relevant context.

How They Compare

What This Category Reveals

These systems are ordered from most-to-least production maturity, based on the complexity of the environment each has been evaluated in and the scale of infrastructure required. ATMem and UI-Mem operate inside containerized Android VMs at scale and have been benchmarked against large proprietary models in real application environments. Memory-R1 and MemChain are evaluated on established long-term memory benchmarks with reproducible baselines and publicly available code. MMPO is the most algorithmically novel, introducing a new intermediate reward signal without changing the memory architecture. ICML is the most architecturally ambitious, adding online policy adaptation during live interactions.

The critical dividing line is between systems that optimize the memory action itself and systems that optimize what surrounds it. ATMem, Memory-R1, and ICML train explicit write decisions. MemChain trains the post-retrieval transformation. MMPO trains intermediate summarization quality. UI-Mem trains what structured experience to carry across tasks. All six treat memory management as a learnable function with a trainable reward, but they operate at different points in the pipeline and with different reward granularity. No single system owns all three steps: write decision, retrieval decision, and post-retrieval transformation. That joint optimization remains an open engineering problem.

The open question this category has not answered is whether a single policy can learn to manage all three decisions simultaneously with a single coherent reward. Most systems here fix at least one decision and train only the others. Joint optimization across the full pipeline may require a fundamentally different training setup, or it may emerge naturally when task horizons are long enough that all three decisions carry measurable downstream impact through the same reward signal.

Limitations and Open Questions

Every system in this comparison carries real caveats. ATMem and UI-Mem evaluate in Android environments using applications present during training; their behavior on genuinely novel app categories and workflow structures is partially tested at best. ICML claims continuous improvement but does not fully characterize how online adaptation behaves over very long interaction horizons, specifically whether recent interactions crowd out earlier, still-relevant memories. Memory-R1 requires only 152 training pairs, but those pairs share the temporal structure of the test set; genuine domain shift is not evaluated.

More fundamentally, none of these systems has been deployed in a production environment where memory accumulates over months of real user interaction at scale. The benchmarks used, LOCOMO, AndroidWorld, RULER-HotpotQA, LongMemEval, are well-designed but do not capture distribution shift, adversarial inputs, or the accumulation of contradicting information that characterizes long-running deployed agents. The RL reward functions are also proxies: a memory policy trained to maximize F1 on held-out QA pairs may systematically ignore information that matters for other tasks the same agent is asked to perform.

There is also no agreed interface. Memory-R1’s CRUD operations, MemChain’s evidence traces, ATMem’s execution state fields, and UI-Mem’s parameterized templates are not interchangeable. A team adopting one approach cannot swap it for another without redesigning the adjacent components. The field is inventing the components before standardizing the interfaces between them.

What This Means for Engineering Teams

If your agent fails at long-horizon tasks because it loses track of intermediate state rather than because it lacks knowledge, ATMem’s execution-state framing is the most directly applicable. The insight that memory should track the current role of a value rather than merely its presence in history applies beyond mobile GUI agents to any workflow where the same data item transitions between pending, processed, and verified states. Teams building production AI agents for structured multi-step workflows will find ATMem’s Constraints and ItemContent fields map naturally to task-tracking logic they are already implementing as custom code outside the model.

If your agent degrades because retrieval floods the answer model with stale or conflicting context, MemChain’s post-retrieval policy is the most targeted fix, and its public code makes it adoptable without retraining the underlying memory construction or answer models. If your agent performs well on short interactions but fails as sessions extend into hundreds of turns, MMPO’s Belief Entropy signal provides a concrete training objective for intermediate summary quality that outcome-only training cannot supply.

For teams building customer-facing conversational agents where personalization should improve over time without human re-labeling, ICML’s online RL approach is the most aligned with that goal. It requires no pre-specified reward tied to a fixed task format, only the ability to collect interaction feedback as sessions complete. The tradeoff is that online training during production introduces risk of policy drift toward recent sessions, which requires monitoring and potentially periodic resets. Teams scaling agentic automation across diverse tasks will increasingly need to choose which memory controller to train, which reward to optimize it with, and how to isolate its behavior for debugging, choices that are today made implicitly by the retrieval library rather than by the engineering team.

Key Takeaways

• ATMem-UI-8B achieves 76.6% success rate on AndroidWorld, outperforming UI-TARS-2-230B by 3.3 points at 1/29 the parameters, by replacing passive memory records with an actively maintained execution state trained via memory-cost-aware RL.
• UI-Mem-8B with inference-time memory retrieval achieves 71.1% on AndroidWorld, surpassing Gemini-2.5-Pro (69.7%) and Seed1.8 (70.7%), using a self-evolving hierarchical experience memory with stratified group sampling that transfers skills across tasks.
• Memory-R1 improves overall F1 by 68.9% over Mem0 on LOCOMO using only 152 training examples, by training a Memory Manager to choose among ADD, UPDATE, DELETE, and NOOP with PPO or GRPO and an Answer Agent to distill relevant memories from 60 retrieved candidates.
• MemChain passes 143.3 evidence tokens to the answer model versus 3,491 for the strongest retrieval baseline, reaching 80.26% accuracy on LoCoMo with a Qwen3-14B answer model, by training a post-retrieval transformation policy with TMPO.
• MMPO maintains 97.1% of peak performance at 1.75M-token contexts and adds +5.47% accuracy at 896K versus standard outcome-based training, by augmenting sparse outcome rewards with dense belief entropy supervision at intermediate summarization steps.
• ICML demonstrates that two co-evolving Actor-Critic agents, one for storage decisions and one for retrieval decisions, can improve response quality and personalization continuously through live interaction rather than requiring offline batch retraining between deployments.

Work With Origins AI

Origins AI builds production AI infrastructure for engineering teams. If your agents lose task context across sessions or fail to improve from accumulated interaction history, talk to our team.