RLTL;DR: Apple’s Approach to Learning From Failed Agent Attempts

Reinforcement learning with verifiable rewards works beautifully when agents occasionally succeed. When every attempt fails, the gradient vanishes, the policy stalls, and no amount of additional sampling helps. Michael Kirchhof et al. (Apple, arXiv:2609.37633, September 2026) published RLTL;DR, a method that extracts a 17-token lesson from each failed rollout, backpropagates through those lesson tokens, and internalizes a task-to-insight mapping into policy weights. The result: 12-13% Pass@1 on tasks where standard GRPO stays at 0%.

The Problem: When No Attempt Ever Succeeds

Standard RLVR runs K parallel rollouts per task, scores each with a verifier, and updates via GRPO (group-relative policy optimization). GRPO computes each rollout’s advantage relative to the group mean. If every rollout in the group receives the same reward, the advantage is zero for all of them. The gradient is zero. The policy does not move. This is the structural failure mode Kirchhof et al. call advantage collapse, also referred to as the learning cliff or exploration inefficiency.

The paper’s experimental setup makes this concrete. Multi-step agentic tasks are formatted as a POMDP with natural-language goal, observation, and action spaces and binary reward. Episodes end when the agent emits an end-of-task action or hits a maximum horizon of 50 actions. Per-step observations show search results and tracebacks. The final observation includes failed assert statements that are only visible to the feedback generator, not the agent during normal operation.

From this task distribution, the authors filter to tasks where Pass@128 = 0. Not Pass@1 equals zero. Pass@128: even with 128 independent attempts, not one succeeds. The base policy is Qwen 3.5 9B Thinking. On these hard tasks, GRPO flat-lines at Pass@1 of 0% to 1%. There is no training signal.

The problem matters because real deployments for agentic AI systems are full of hard tasks. The tasks where agents reliably succeed are the ones that do not need training. The tasks worth improving are the ones agents currently cannot solve.

How RLTL;DR Works

RLTL;DR has two components. Each one is a minimal change to the standard GRPO loop. Combined, they move Pass@1 from 0% to 12-13% at evaluation time, when no insights are in the model’s context.

Sequential Exploration With Self-Generated Insights

Standard GRPO samples K rollouts in parallel. RLTL;DR replaces this with sequential rollouts. The difference is that each attempt can observe the result of the one before it.

Here is the sequence for a single task in training. The agent attempts the task and produces trajectory tau_1. If it fails, the policy is given the previous trajectory and the error message. It generates a TL;DR insight f_1: a single-sentence summary of what went wrong and what to change. Something like “Remember to paginate search results.” This insight is roughly 17 tokens long.

The next attempt is conditioned on the task plus all insights collected so far. Attempt two has the benefit of the lesson from attempt one. Attempt three has the lessons from attempts one and two. The process continues for K attempts total. The K successive attempts are treated as a single GRPO group for advantage computation.

The insight insertion is not unconditional. The authors use a 50% success boundary: insights are inserted only when fewer than half of the preceding attempts in the group succeeded. This keeps the agent in a productive zone. When it succeeds often, it does not need handholding. When it never succeeds, the insights give it a foothold.

The insight generation prompt asks the agent to think through the failed trajectory, analyze the error, and output a single-sentence summary of what to improve. The policy generating insights is the same policy being trained, with the same evolving weights. There is no separate teacher model, no gold solutions, and no reference implementations. The agent critiques itself.

This mechanism solves the exploration problem. On tasks where Pass@128 was 0 under standard sampling, sequential rollouts with insights achieve Pass@k of 14-59% on the tasks seen during training, meaning at least one success appears in the group. The GRPO advantage can now be computed from real variation in outcomes. The learning cliff is avoided.

But there is a catch. At evaluation time, insights are not in the context. The agent must solve tasks without them. With insights in context during evaluation, Pass@1 reaches 14-31%. Without insights in context, it falls to 12-13%. Better than 0%, but the sequential exploration alone is not the full story. That is what the second component handles.

Backpropagating the Insight Tokens

The second component is four words: activate the backpropagation mask on insight tokens.

During training, the insight tokens are present in the context as user messages between attempts. Standard GRPO would apply a training loss only to the rollout tokens the policy generated. The insight tokens, which the policy also generated but which appear in the context as prior messages, would be masked out of the gradient computation.

RLTL;DR unmasks them. The training objective becomes a GRPO loss on rollout tokens plus an SFT loss on the roughly 17 insight tokens per insight. That is the full change.

What does backpropagating through 17 tokens accomplish? It trains a mapping: task features to “something to keep in mind.” The insight tokens are never generated at test time. At test time, there is no separate insight-generation step; the agent just sees the task and produces a response. But the weights have learned that certain task structures call for certain classes of attention. The 17-token gradient does not need to teach the full solution; it needs to tilt the model toward the right approach, and the smoothness of backpropagation generalizes that tilt across similar problem types.

The authors describe this as internalizing a “task to insight” mapping into policy weights. The insight is the compressed lesson; the internalization is the mechanism by which that lesson generalizes beyond the specific task it was extracted from.

The ablation result is striking. If you turn off the GRPO loss entirely and keep only the SFT loss on insight tokens, performance is nearly identical to the full RLTL;DR method. The backpropagation on rollout trajectories is not the key ingredient. The 17-token gradient on insights carries most of the signal. This reframes what is happening: RLTL;DR is closer to a targeted self-distillation on compressed lessons than it is to standard reinforcement learning on trajectories.

SFTL;DR: The Distilled Version

The ablation pointing to insight-token backpropagation as the key ingredient leads naturally to a question: do you need the rollouts at all?

The answer is mostly no. SFTL;DR is a reduction of RLTL;DR to pure supervised learning. It trains only on (task, insight) tuples. No rollouts are shown during training. No backpropagation on trajectories. Just 4,000 (task, insight) pairs. The result recovers almost the full performance of RLTL;DR, and it also matches classical SFT on full rollouts despite using only the compressed lesson rather than the complete trajectory.

The authors describe this as “a compacted training paradigm of the form ‘on this sort of task, keep this sort of thing in mind.’” The insight is a sufficient representation for training the generalization. The full rollout adds noise, length, and compute without proportionate benefit to the learned policy.

This is a practical finding for anyone considering the model selection and fine-tuning question. If you can generate (task, insight) pairs, either through the sequential rollout mechanism described above or through other means, you do not need the full rollout data. You need the compressed lesson. That is a substantial difference in data efficiency and training cost.

SFTL;DR also demonstrates something more general: models can apply knowledge in a different format than they were trained on. The policy is trained to predict a 17-token insight given a task description. At test time, it generates full multi-step rollouts. The gap between those formats does not prevent the training signal from transferring. The generalization happens through the semantic content of the lesson, not through format matching.

Results

The evaluation uses challenging tool-calling and coding datasets filtered to Pass@128 = 0. These are tasks where standard sampling and training methods produce no signal at all.

Standard GRPO on these tasks: Pass@1 of 0% to 1% across the board. The training curve is flat.

RLTL;DR with insights in context during training: Pass@1 of 14-31%. This is the ceiling with explicit reasoning support available.

RLTL;DR at evaluation time with no insights in context: Pass@1 of 12-13%. This is the number that matters for deployment. The policy has internalized enough from the insight tokens that it can solve 12-13% of tasks it never solved before, without any in-context guidance at test time.

The privilege condition matters here. Insight quality depends on access to verifier feedback. The policy sees the final observation including failed assert statements, which are only visible to the feedback generator and not to the agent during normal task execution. When access to that privileged verifier information is removed, insight quality drops sharply. The feedback the agent critiques is not the feedback it saw during the failed attempt; it is the diagnostic output from the verifier that only becomes available post-hoc. That information is what makes the lessons specific and actionable rather than generic.

Limitations and Open Questions

The method depends on verifier access. For tasks without a formal verifier, assembling the right feedback signal to generate useful insights becomes the primary engineering challenge. The 17-token insight is only as good as the diagnostic information the policy can see when generating it.

Sequential rollout collection is slower than parallel. The K attempts for a single task must run in order rather than concurrently. This increases wall-clock training time. The paper uses synchronous rollout collection; the tradeoff between training efficiency and sequential exploration is not fully characterized in the paper.

The 50% threshold for insight insertion is a fixed hyperparameter. The authors do not analyze sensitivity to this value or whether a learned threshold would perform better. It is a reasonable default, but its robustness across different task distributions and model sizes is uncharacterized.

The evaluation is on two task categories: tool-calling and coding. Both have formal, automatable verifiers. The degree to which RLTL;DR generalizes to tasks with messier evaluation criteria, such as open-ended planning or multi-party negotiation, is an open question. The insight generation step specifically benefits from clean error messages and failed asserts; natural-language-only feedback may produce noisier lessons.

What This Means for Engineering Teams

The practical takeaway is about what you need to make training work on hard tasks. You do not need gold solutions. You do not need a stronger teacher model. You need a verifier that can tell the agent what went wrong, a sequential rollout mechanism that lets the agent see its own failure before the next attempt, and a backpropagation mask covering the insight tokens. That is a feasible pipeline for many real production settings.

Teams working on agentic AI systems that hit a training ceiling on difficult tasks should pay attention to the SFTL;DR result specifically. If you have a task corpus with a working verifier, you can use the sequential rollout mechanism to generate (task, insight) pairs on tasks where your agent fails, then fine-tune on just those pairs. 4,000 examples is a manageable dataset. The full rollout data is not required.

This connects to wider work on reinforcement learning training techniques where the structure of the training signal matters as much as its presence. The GRPO failure mode on zero-success tasks appears whenever the task distribution is hard enough that parallel sampling finds nothing. Sequential exploration with self-generated lessons is one answer to that failure mode. The key architectural choice, putting the compressed insight on the gradient path rather than just in the context, is what distinguishes RLTL;DR from simply adding a chain-of-thought step before each retry.

For teams doing model selection and fine-tuning on agentic tasks, the SFTL;DR result has a direct workflow implication. Generate failures, extract lessons, train on lessons. 4,000 examples is a manageable dataset. The full rollout data is not required, and compressed insights recover nearly identical performance.

Key Takeaways

• GRPO produces zero gradient when all K rollouts in a group receive the same reward. On tasks where Pass@128 = 0, this means no learning happens at all. RLTL;DR’s sequential rollout design is the fix for this structural problem.
• The key ingredient is not the GRPO loss on rollout tokens. It is the SFT loss on the 17-token insight tokens. Turning off GRPO entirely and keeping only insight-token backpropagation produces nearly identical results, which means the compression of the lesson is what generalizes, not the reinforcement signal on the trajectory.
• SFTL;DR shows that 4,000 (task, insight) tuples, with no rollout data at all, recovers almost the full performance of the complete method. This is a dataset-efficient path to training agents on hard tasks, provided you have a verifier that can generate informative failure feedback.
• At evaluation time, without any insights in context, RLTL;DR-trained agents achieve 12-13% Pass@1 on tasks where the untrained baseline achieves 0%. The internalization is real, not an artifact of in-context assistance, and it generalizes across similar task structures through the smoothness of backpropagation.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If your agents are hitting a training ceiling on hard tasks where standard RL produces no gradient and you need a path to improving performance without gold-labeled solutions or teacher models, talk to our team.