Six MoE Inference Runtimes Serving Expert Weights Directly From SSD

On September 16, 2026, two independent research groups published papers on the same idea on the same day: store the complete expert pool of a large MoE model on NVMe SSD, and serve only the small active working set into RAM and VRAM at inference time. Edge0, from AutoArk, reported a 35B-class model running at 20.4 tokens per second inside 2.9 GiB of peak active memory on a Mac Mini M4 Pro. SSD-LLaMA, from HKUST and WiCi AI, reported achieving over 1 token per second on a trillion-parameter model using one RTX 5090 and 32 GB RAM. Their simultaneous arrival is not a coincidence. A question that seemed settled, namely that large models simply require proportionally large memory, now has a different answer for sparse MoE architectures.

Why This Comparison Is Timely

The two September 16 papers arrived in the same week as the BigMoMo paper from Peking University, submitted September 13, which applied the same principle to mobile NPUs and reported a 4.83x speedup over on-demand autoregressive offloading. Three serious storage-native MoE runtimes appeared in four days. That concentration of simultaneous, independent work points to a constraint that has become acute: frontier MoE models, even after 4-bit quantization, exceed the RAM and VRAM of most local hardware by a factor of three to ten.

The hardware context matters. PCIe 5.0 NVMe SSDs now reach 14.8 GB/s sequential bandwidth, according to SSD-LLaMA’s hardware survey, which exceeds the 12.8 GB/s of single-channel DDR3-1600. That convergence did not exist three years ago. Flash storage is no longer orders of magnitude behind DRAM in sequential throughput. It is still slower for random small reads, but MoE expert weights arrive in predictable, relatively large blocks once routing is known. The gap that made SSD inference impractical has narrowed to the point where several teams, independently, concluded it was worth closing.

The Shift: Load the Whole Model, Then Run It, Becomes Stream Only the Active Expert Working Set

Standard MoE inference starts with a requirement: the complete weight tensor must fit in host memory or VRAM before the first token generates. A 35B-class MoE model at 4-bit quantization occupies roughly 19.5 GB. A typical consumer machine with 24 GB unified memory, once the OS and KV cache take their share, has almost nothing left.

The storage-native approach inverts that requirement. Only the experts routed to by the current token, or predicted to be routed to by the next token, need to be resident. The rest stay on SSD. On a Transformer with 40 MoE layers and top-4 routing, that means at most 4 experts times 40 layers, or roughly 7 MB of active expert weights per decode step. Not 19.5 GB. The question each system answers differently is how to get those 7 MB from disk to the accelerator fast enough that decode throughput does not collapse. To understand how MoE expert routing works at the architecture level, the linked post covers the fundamentals.

MoE-Infinity: Multi-Tier Expert Cache With Activation-Aware Replacement

MoE-Infinity, published by the EfficientMoE team and available at github.com/EfficientMoE/MoE-Infinity, established the broader architecture that later systems build on or respond to. The system organizes expert weights across three tiers: GPU memory (VRAM), host memory (DRAM), and SSD storage. A coordinating cache layer decides which experts to hold at each tier based on activation frequency and reuse distance, rather than simple LRU or FIFO.

The core insight in MoE-Infinity is that expert access patterns are not uniform. Across a model with hundreds of experts, a small subset consistently handles the majority of tokens in any given input domain. The activation-aware cache exploits this skew by retaining hot experts in VRAM and cold experts in host memory, falling back to SSD only for true cold misses. The system treats the SSD as the lowest-cost tier in a three-tier expert pool, not as a last resort after RAM runs out.

MoE-Infinity is the system that Edge0, SSD-LLaMA, and FlashMoE all cite as an existing baseline to surpass or differentiate from. It popularized the idea that expert offloading could be a first-class inference mode rather than a degraded fallback. Its limitation, as Edge0’s authors note, is that it moves the weight footprint to a different location rather than fundamentally shrinking what needs to be resident at any instant. Later systems target exactly that residency bound.

FlashMoE: Learned Cache Replacement That Beats Recency and Frequency Alone

FlashMoE, from a team at KAIST, submitted January 22, 2026 (arXiv:2601.17063), focuses on a specific problem in SSD-backed expert serving: LRU and LFU cache replacement policies are provably bad for expert routing patterns. The paper measures that when LRU evicts an expert, that expert gets re-fetched within the next 5 decode steps 34.2% of the time, compared to Belady’s optimal policy at 0.1%. LRU is evicting experts that are about to be needed again.

The reason is structural. Expert routing creates two failure modes for LRU. First, experts routed to at a given token are immediately recency-fresh, so they survive in cache even after they will not be needed again, pushing out other experts. Second, popular experts that happen not to be routed to for a few consecutive steps get evicted by recency before they recur, causing expensive re-fetches. When the paper compares LRU and LFU eviction choices head-to-head on Qwen3-30B-A3B, LRU makes the better decision only about 56% of the time, and LFU does roughly equally poorly on the scenarios LRU handles badly.

FlashMoE replaces both with a lightweight ML-based cache policy trained to approximate Belady’s optimal replacement. The learned policy combines recency and frequency signals into a predicted next-access time, choosing which expert to evict based on the longest horizon to next use rather than on recency or frequency alone. On real desktop hardware, FlashMoE reports a cache hit rate improvement of up to 51% over LRU and LFU, translating to a 2.6x speedup over existing MoE inference systems in a desktop environment. Expert loading accounts for over 70% of total decoding time in their setup, making cache hit rate the dominant performance lever.

SpecPrefetch: A Lightweight Adapter That Predicts Without Changing Routing

SpecPrefetch, submitted June 24, 2026 (arXiv:2607.24787), takes a deliberately conservative position on routing. Where Edge0 makes its predicted route become the actual route, SpecPrefetch adds a lightweight shared adapter that predicts which experts the next layer is likely to need, but treats that prediction as advisory only. The model’s native router remains authoritative. Predicted experts are asynchronously prefetched from slower memory in the background while the current layer executes.

The design preserves exact model behavior. No token ever runs through different experts than the original router would select. The adapter’s predictions can be wrong without affecting output quality, because a wrong prediction simply means the prefetch was a wasted read, not that the computation used incorrect weights. That property matters for production deployment, where output reproducibility relative to the base model checkpoint is a requirement in many settings.

SpecPrefetch is parameter-efficient in a specific sense: the adapter is shared across layers rather than one per layer, reducing the training and storage cost of adding prefetch capability to an existing model. The tradeoff is that the shared adapter cannot specialize to individual layers’ routing patterns as precisely as a per-layer head can. The system demonstrates that conservative, non-routing-altering prediction is a viable point on the design space, relevant when model fidelity requirements prohibit changing the base routing path.

SSD-LLaMA: A Three-Tier Native Pipeline From NVMe to GPU

SSD-LLaMA, from HKUST and WiCi AI, submitted September 16, 2026 (arXiv:2609.18110), addresses a problem that earlier offloading approaches mostly ignored: even when SSD bandwidth is adequate, most model layouts are not organized for expert-granular access. Standard checkpoints scatter one expert’s tensors across multiple file regions. Memory-mapped access then causes page faults and fragmented reads that underuse the available SSD bandwidth.

SSD-LLaMA reorganizes each expert’s tensors into a single aligned, contiguous block before serving begins. At decode time, the system issues concurrent direct reads for each needed expert block rather than sequential memory-mapped page reads. As each read completes, the expert block transfers through pinned RAM to its reserved VRAM slot without waiting for the other reads to finish. The system also pairs a CUDA rANS decoder for lossless on-GPU decompression, reducing SSD-to-GPU data volume for cold experts that cannot be retained in either RAM or VRAM.

The three-tier hierarchy operates as a single coordinated pipeline: SSD holds the complete expert pool, RAM caches loaded experts by observed frequency and recency, and VRAM retains the hottest working set for immediate GPU access. The system separates weight residency from execution placement, assigning expert computation to the GPU whenever doing so avoids CPU-GPU synchronization stalls, rather than binding every RAM-resident expert to CPU execution. Across three frontier MoE model families, SSD-LLaMA improves prefill token rate by 1.52x to 4.19x and decode token rate by 2.10x to 15.58x over evaluated baselines. For Kimi-K2.7-Code, a model with more than one trillion parameters at 500 GB under 4-bit quantization, the system achieves a decode rate above 1 token per second using one RTX 5090 and 32 GB of RAM. That is a model whose VRAM-only storage would cost roughly $32,000 at GPU memory pricing.

BigMoMo: Speculative Decoding as an I/O Scheduling Window

BigMoMo, from Peking University and Tianjin University, submitted September 13, 2026 (arXiv:2609.14643), targets a different deployment context: mobile NPUs with UFS flash storage, where sequential SSD bandwidth is around 4 GB/s and access latency is substantially higher than desktop NVMe. On mobile, the per-token sequential coupling between routing and expert loading is especially costly. Each expert transfer serves few tokens before execution moves on, leaving the NPU starved.

BigMoMo’s core mechanism is using the multi-token verification window of speculative decoding as an I/O scheduling opportunity. Standard speculative decoding generates several candidate tokens with a draft model, then verifies them all with the target model in a single forward pass. BigMoMo observes that this verification window exposes the expert demand for multiple tokens simultaneously, enabling three things that per-token loading cannot achieve. Multiple speculative tokens may route to the same expert, so that expert’s weights serve several tokens from one load. Multiple expert requests within one verification round can be batched and reordered into more contiguous flash reads. And experts from different memory hierarchy levels can be moved and computed in a cross-stage overlap, batching NPU-ready experts together rather than waiting for all experts before starting execution.

The system adds three mechanisms on top of speculative decoding: reuse-aware expert movement admission, which prunes low-value speculation branches and low-utility expert activations based on acceptance statistics and data movement cost; runtime-adaptive on-flash expert organization, which incrementally reorganizes flash placement according to observed co-loading patterns, converting fragmented accesses into sequential reads; and cross-stage movement-computation scheduling, which tracks expert residency across movement stages and dynamically batches ready experts to overlap later loading with NPU computation. Across four MoE models and five benchmarks on two mobile platforms, BigMoMo achieves a mean decoding speedup of 4.83x over on-demand autoregressive offloading, and a maximum 1.82x time-per-output-token reduction over conventional speculative MoE baselines, supporting models up to 30B parameters.

Edge0: When the Prediction Becomes the Routing

Edge0, from AutoArk, submitted September 16, 2026 (arXiv:2609.18063), makes the most aggressive architectural choice in this space. Its core insight is that the information dependency causing disk stalls is: layer N+1’s expert selection depends on layer N’s output, which does not exist yet when layer N+1’s expert reads would need to start. Every other system in this comparison accepts that dependency and tries to hide it with caching, prefetching, or batching. Edge0 removes the dependency by making the prediction the routing itself.

A per-layer head called the prerouter predicts the next layer’s routing one full token ahead, using the current layer’s post-attention norm output and the past two routing decisions as features. The head produces expert logits that pass through the model’s own routing math (softmax-top-k or sigmoid-group, depending on the model family). The resulting routing replaces the native router’s output entirely at decode time. Because the predicted expert set and the routed expert set are identical by construction, there is nothing to drop and no fallback needed. The SSD read that fills the staged slots for layer N+1 starts overlapping the current forward pass of layer N. On the 16 GB MacBook M2 test setup (where the 18.4 GiB checkpoint does not fit in physical RAM), this reduces the main thread’s blocked time on expert loads from 244.0 ms per step to 101.9 ms per step at K=4, yielding a decode throughput gain of 82%.

The quality cost of prediction-as-routing and 4-bit quantization is recovered by a distilled LoRA adapter, served unmerged as a parallel delta alongside the frozen int4 base. Merging the adapter into the 4-bit weights and re-quantizing destroys most of the effect: only 34% of the adapter’s contribution survives on an attention projection after re-quantization, and only 18% at the logit level. Serving it unmerged costs 42 MB of adapter weights and no measurable decode time. On the 35B production tier at K=4, Edge0 achieves 20.4 tok/s decode and 2.9 GiB peak active memory on a Mac Mini M4 Pro 24 GB, against 3.9 tok/s and 18.2 GiB occupied for the same weights loaded fully resident. The mean quality gap across five OpenCompass benchmarks is 3.9 points below the fp16 base. The framework, checkpoints, and adapters are open source at github.com/Edge0-AI/Edge0.

How They Compare

What This Category Reveals

The six systems span a spectrum from conservative to aggressive in how much they alter the model’s own computation. MoE-Infinity and SSD-LLaMA treat routing as untouchable and focus entirely on storage and I/O engineering. FlashMoE adds a learned cache policy but does not touch the router. SpecPrefetch predicts routing but keeps the native router authoritative. BigMoMo coordinates I/O scheduling around speculative decoding’s multi-token window without modifying routing. Edge0 replaces the router entirely with a trained predictor. That ordering, from no model change to deep routing intervention, roughly corresponds to increasing throughput gain and increasing complexity of deployment and quality assurance.

The entries are ordered from most-to-least established because MoE-Infinity set the baseline architecture and the more recent papers respond to it explicitly. What the comparison reveals is that the storage I/O problem has multiple solutions at different points on the design tradeoff surface. A team that cannot accept any routing modification has SSD-LLaMA, FlashMoE, and MoE-Infinity as options with meaningful throughput gains. A team willing to train and maintain routing-prediction components can get qualitatively different memory footprints, as Edge0 demonstrates with its 2.9 GiB peak active memory figure against a 19.5 GB checkpoint.

The critical ingredient separating the faster systems from simpler offloading is predictive scheduling. Every approach that achieves substantial gains, whether through prerouter prediction, speculative-decoding windows, or learned cache replacement, creates an information advantage: it knows what experts will be needed slightly before they are needed, and uses that knowledge to start I/O early. The open question the category has not answered is how these systems behave under concurrent multi-user load. Every measured result here is single-user, single-request. Concurrent requests produce overlapping, potentially conflicting expert working sets. How expert caches, admission policies, and prefetch predictions perform under that condition is not yet reported by any of the six systems.

Limitations and Open Questions

Model semantics: three of the six systems change nothing about the model’s output. Edge0 does not. Its prerouter replaces the native router, and the quality gap is real: 6.1 points on AIME 2026 for the 35B tier, and 10.0 points for the 8B tier, compared to narrow gaps on most other benchmarks. The recovery LoRA closes most of the gap on general tasks but leaves long-chain reasoning visibly affected. Teams choosing Edge0 accept a quality tradeoff in exchange for the memory footprint reduction.

Scale and hardware coverage: most benchmark results come from single-GPU desktop or mobile setups. SSD endurance under sustained expert streaming, tail latency under cache misses, and performance under multi-user batching are not yet characterized. BigMoMo’s Qualcomm Hexagon NPU results do not transfer directly to other mobile SoC families. FlashMoE’s desktop results come from a user-grade setup that may not represent either high-end desktop or mobile deployments accurately. Edge0’s MLX backend limits it to Apple Silicon; the paper explicitly notes the CUDA slot is “architecture, not code” as of the current release.

SSD endurance is a practical concern that none of the papers quantify. A system decoding at 20 tok/s with 7 MB of expert reads per step moves roughly 140 MB per second through the SSD continuously. Modern consumer NVMe drives specify TBW (terabytes written) endurance ratings but not terabytes read; read endurance is generally much higher than write endurance, but sustained high-frequency reads at inference scale have not been studied in this context.

What This Means for Engineering Teams

The immediate practical implication is that model size is no longer a flat VRAM requirement for MoE architectures. The relevant question shifts to: what fraction of the model must be resident at any instant, and how predictable is the next expert working set? For teams evaluating these systems, the comparison dimensions that matter most are peak active memory, routing fidelity (whether model outputs match the base checkpoint), prefetch prediction mechanism, and SSD bandwidth requirements. Understanding how the inference memory hierarchy works at the kernel level helps interpret the tradeoffs between these systems.

Teams building on-premise inference for large sparse models that cannot afford GPU clusters should look at SSD-LLaMA and Edge0 first, because both target consumer hardware and both have open implementations. Teams deploying on Qualcomm mobile SoCs should examine BigMoMo’s approach, particularly its runtime-adaptive flash reorganization, which converts fragmented random reads into sequential reads during execution. Teams unable to retrain any model component should use SSD-LLaMA or FlashMoE, both of which treat the checkpoint as read-only. Compression techniques like quantization remain a prerequisite for all six systems, since they reduce the SSD footprint of the complete expert pool before storage-native serving begins.

The deeper engineering implication is that a production storage-native MoE inference stack looks more like a virtual memory system than a standard model server. Token arrives, router predicts (or prerouter predicts), expert scheduler decides what to fetch or evict, I/O pipeline delivers weights to accelerator, compute executes, cache policy updates. That pipeline has its own tuning parameters, failure modes, and monitoring needs separate from standard LLM serving. Teams planning to use these systems in production should allocate engineering capacity for storage scheduling logic, not just for model deployment.

Key Takeaways

• Edge0 runs a 35B-class MoE at 20.4 tok/s inside 2.9 GiB of peak active memory, against 18.2 GiB for the same weights loaded fully resident, a reduction factor of over six.
• SSD-LLaMA achieves over 1 token per second on a model exceeding one trillion parameters using a single RTX 5090 and 32 GB RAM, by reorganizing experts into contiguous on-disk packs and using concurrent direct reads.
• BigMoMo treats speculative decoding’s multi-token verification window as an I/O scheduling opportunity, achieving 4.83x average speedup over on-demand mobile autoregressive offloading for models up to 30B.
• FlashMoE’s learned cache replacement achieves a 51% higher cache hit rate than LRU and LFU, because expert routing creates reuse patterns that recency and frequency signals alone cannot capture.
• The key architectural distinction is whether a system predicts routing and uses that prediction advisorily (SpecPrefetch, BigMoMo) or makes the prediction authoritative (Edge0). The latter achieves dramatically lower residency at the cost of model fidelity and a retraining requirement.
• All six systems report single-user results. Multi-user concurrent serving, SSD endurance under sustained inference load, and tail latency under cache misses remain uncharacterized across the entire space.

Work With Origins AI

Origins AI builds production AI infrastructure for engineering teams. If your team is evaluating large MoE models for on-premise inference and needs guidance on storage-native serving architectures, talk to our team.