mini-AGI Slashes Catastrophic Forgetting to 0.007 Nats on 8 GB GPU

On September 21, 2026, a project called mini-AGI appeared on GitHub with a result that is easy to dismiss and harder to explain away: after reading 524,000 characters of chess text on a single consumer GPU, the model retained 99.84% of its prior knowledge on seven unrelated subjects. No replay buffer. No task labels. No separate fine-tuning pass. The mechanism is one number in a config file.

The project author, volotat, is explicit that this is a toy-level experiment, not a frontier model. The trained weights are not yet published because the first training pass is still running. What mini-AGI contributes is a concrete, reproducible systems design for continual learning on modest hardware, with a measurement that challenges a commonly held assumption about where the catastrophic forgetting problem actually lives.

Why Catastrophic Forgetting Has Blocked Personal AI

Every language model available today is a frozen artifact. You can run inference on it, fine-tune the edges of it, or attach retrieval to compensate for what it does not know. You cannot train one from scratch on your own hardware and keep training it as you use it. The moment you try sequential learning on a single stream of data, the model overwrites what it previously learned. Reading chess wipes out what the model knew about code. Reading code wipes out stories. The standard remedy is a separate fine-tuning job against a frozen base, which is not training but adjustment.

mini-AGI is designed around three constraints that make this problem tractable on a laptop with 8 GB VRAM. First, the weights live on disk and only the working set occupies GPU memory, so total parameter count is bounded by free disk space rather than by VRAM. Second, the model never has to stop learning. Third, the input alphabet is the 256 possible byte values, so there is no tokenizer and no data type that requires a new vocabulary. Any file is already written in the model’s language.

How the Architecture Works

A character in mini-AGI does not pass through a fixed stack of layers. It passes through two dense prelude blocks and then through one recurrent block applied up to 24 times, with each application choosing its own set of experts from a shared pool. The recurrent block can fire up to 26 times per character total.

Depth is adaptive. A halting head scores every character at every row and stops computation as soon as another row would not change the prediction. The implementation follows the PonderNet recipe: during training, every depth is computed and weighted by its halting probability, so the halting head learns through those weights rather than through a separate supervision signal. In practice, the model uses between 4 and 14 rows per character, against a ceiling of 24. Easy characters halt early. Hard ones run longer. Writing costs more depth than reading: about 9.9 rows per character during generation versus 8.0 rows per character during inference on the same subject.

Routing happens per block-application, not per character. Each of the up to 26 applications picks its own top-8 experts from the pool. One character therefore touches far more of the pool than “top-8” suggests, and the same expert can appear at multiple depths. No expert is assigned a subject. Soft top-k routing distributes capability across the pool without labels, and a character can combine fragments from several experts.

The current documented run has 540.1M parameters across 169 experts, with approximately 109M parameters resident in VRAM at any given time. The active working set fits on an 8 GB GPU; the rest of the model lives on disk.

Expert Paging: Disk as Parameter Store

Every expert is a file on disk that holds its weights and its Adam optimizer moments. Three tiers sit above that disk layer: a least-recently-used RAM cache, then the VRAM resident set.

Before each training chunk, the model decides which experts the upcoming text actually needs. Demand is scored on the hidden states the call sites routed on during the previous chunk, not on raw input embeddings. Using embeddings would cause every subject to ask for the same experts; routing on actual hidden states respects what the text is computationally asking for.

Two rules hold through every swap. Adam’s moments travel with the expert, not with the VRAM slot it occupied. Leaving moments behind would hand one expert’s momentum to whatever took its place, and training would appear healthy while every swapped expert inherited a stranger’s optimization history. An expert already on the card also stays in its slot regardless of demand rank, because stability in the working set reduces unnecessary loads.

A hysteresis rule governs displacement: a candidate expert has to beat a resident by a configured margin before it displaces it, and a recently loaded expert is safe for a minimum dwell period. This keeps the working set from churning on transient routing signals.

How Continual Learning Works: The Trunk Learning Rate

Training on a single stream, one subject at a time, is the standard recipe for catastrophic forgetting. If you read half a million characters of chess at the standard expert learning rate, the model’s loss on the seven other subjects climbs from 1.12 nats to 3.73 nats. Most of what it knew is gone.

The mechanism mini-AGI uses to prevent this is the trunk learning rate. The trunk is the part of the model every character passes through regardless of routing: embeddings, attention, routers, and the halting head. It carries 97.6% of the squared gradient norm, which means it is the part of the model that moves most during training.

Running the trunk at 0.1x the experts’ learning rate takes forgetting from +2.2300 nats to +0.0067 nats. The table from the README captures this cleanly:

The gap between 50.68% and 99.84% is one hyperparameter. The gap between 42.88% and 50.68% is the expert pool, which contributes less than one-quarter of the total effect. This corrects what the project itself describes as an earlier misreading: the expert pool is not what prevents forgetting. Even with the working set frozen and 93 of 136 experts receiving no gradient at all, the model still collapsed on unread subjects. Preserving most of the weights is not sufficient. What matters is which part of the model moves and at what rate.

The README offers a second finding about the nature of the forgetting. Damage the model badly by reading chess at equal learning rates, then read all seven subjects again. Three-quarters of the lost knowledge returns in 131,000 characters, against the roughly 50M characters it took to learn those subjects originally. Relearning is 380x faster than initial learning. The project’s interpretation: “catastrophic” describes how the loss curve looks at the bottom, not what happened to the underlying weights.

Results: What the Numbers Show

At the time of the September 21 publication, the model had read 409.0M characters and grown to 175 experts. Current benchmark numbers, which the project notes are moving as training continues:

• Held-out loss across all 8 subjects: 0.7980 nats/char (1.1512 bits/byte)
• Training loss: 0.6137 nats/char (0.8854 bits/byte)

Per-subject held-out loss shows the expected gradient: chess (0.511 nats/char) leads because the model has read the most chess, while wikipedia (1.275 nats/char) trails as the most structurally complex subject. Code sits at 0.679 nats/char and reasoning at 0.747 nats/char.

The forgetting probe during the live trunk-at-0.1x run shows that chess improved by 0.013 nats/char over the 524,000-character probe, while the seven unread subjects stayed within ±0.02 nats of their starting values. Only 54 of 136 experts received any gradient during the chess probe: 60% of the model was structurally untouched because routing never selected it.

The learning rate uses no schedule. A cosine schedule assumes the run ends, which is false for a model designed to never stop training. Instead, a controller watches held-out loss and adjusts the rate in both directions: clear improvement earns a small rate increase, stagnation eases it down, and a confirmed held-out spike steps it back up.

Limitations and Open Questions

The project is a self-described toy-level experiment. The trained weights are not published because the first pass through the corpus has not finished; the author estimates a couple of weeks from the September 21 post date. The evaluation is narrow: eight subjects defined by the author, tested by the author, with no independent reproduction.

The same-run-twice variability on CUDA is about 0.014 nats, and the project suggests treating 0.03 nats as the threshold for a real difference. The held-out chess probe result of +0.0067 nats sits near that noise floor, which means the forgetting number should be read as “near zero” rather than as a precise measurement.

The model generates grammatically correct, on-topic text but repeats itself at 243M characters read. “What it produced, continuing a story about a cherry tree: They worked together and saw their favorite shore. One day, they wanted to play with their favorite shore.” Coherence at scale is an open question.

The expert pool grows by recombination, taking whole hidden units from several existing experts. Whether this scales to much larger pools, or whether routing instability grows with pool size, is not addressed. Related academic work, including arXiv:2609.06986 on long-horizon continual learning, finds that naive sequential fine-tuning retains only 1.2% of prior performance, while combining multiple regularization anchors raises retention to 34.9% but across a very different architecture. mini-AGI’s approach is not benchmarked against these methods.

What This Means for Engineering Teams

The immediate practical contribution is a design pattern, not a production model. mini-AGI demonstrates that a disk-paged expert pool plus a slow-moving trunk can decouple a model’s total capacity from its hardware constraints and eliminate the training-then-deploy cycle at small scale.

For teams building AI models for applications that need to adapt to user behavior over time, the architecture raises a question worth asking: how much of the current fine-tuning infrastructure exists because of the assumption that training and inference must be separate events? If that assumption is questioned, the infrastructure that supports the separation, including checkpoint management, scheduled fine-tuning jobs, and rollback systems, becomes a design choice rather than a necessity.

The paging design has a parallel in how operating systems handle memory. The model’s total capacity lives in storage; the GPU holds only what the current task needs. This is familiar from database buffer pools and virtual memory systems. Applying the same pattern to neural network weights is not new as an idea, but mini-AGI provides a working implementation with a concrete memory management policy and an optimizer that travels with its weights.

The anti-predictive gate finding is practically useful regardless of this specific model. The project reports that the smallest routing gates belong to the busiest experts, and the highest gates belong to experts that nothing has asked for in hundreds of thousands of segments. If you are building a MoE system and using gate values to identify capacity to prune, this is a reason to validate that assumption before acting on it. Teams working on LLM model compression should note that gate magnitude is not a reliable proxy for expert utilization.

Key Takeaways

• Running the model trunk at 0.1x the expert learning rate reduces catastrophic forgetting from +2.2300 nats to +0.0067 nats across seven subjects after 524,000 characters of single-subject training.
• The expert pool is not the primary anti-forgetting mechanism: freezing the working set entirely while keeping trunk and expert LR equal still produced +2.5871 nats of forgetting.
• Expert weights and Adam moments live on disk as files; only the working set occupies VRAM, so total model capacity is bounded by disk space rather than GPU memory.
• The routing gate is anti-predictive: smallest gates identify the most-used experts, making gate magnitude an unreliable pruning signal in MoE systems.
• Lost knowledge recovers approximately 380x faster on re-reading than it took to learn initially, suggesting forgetting damages weight organization more than weight values.
• The model, the training code, and the serving stack are MIT-licensed; the weights are not yet published as of September 2026.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If your team is evaluating continual learning or on-device model adaptation that keeps improving after deployment, talk to our team.