Stanford’s 1.5-SPSA Trains OPT-30B Without Backpropagation
On September 29, 2026, researchers Francois Chaubard, Mykel J. Kochenderfer, and Chris RΓ© at Stanford University published two papers (arXiv:2609.38095 and arXiv:2609.37899) that make zero-order optimization competitive with backpropagation on real fine-tuning benchmarks, not as a curiosity, but with a mechanism that explains when it works and by how much. The first introduces 1.5-SPSA, which beats backpropagation accuracy on SST-2 in 70 steps while training OPT-30B in roughly 60 GB of GPU memory instead of 600 GB. The second introduces SOMA, an architecture designed to reduce gradient estimation noise as ensemble size grows. Together they make the case that memory cost, not algorithmic quality, has been the main obstacle to zero-order training at scale.
Why Backpropagation’s Memory Cost Is a Real Constraint
Training OPT-30B with Adam at batch size 8 and sequence length 2048 requires approximately 600 GB of GPU memory: intermediate activations for every layer on every forward pass, plus 32-bit optimizer states that double or triple the parameter footprint. At 600 GB, you need a multi-node cluster with expensive high-bandwidth interconnects.
Zero-order optimization sidesteps this entirely by training in inference mode. No stored activations, no gradient tensors, no Adam states. The same OPT-30B in inference mode occupies roughly 60 GB, a 10x reduction. The cost is gradient quality: instead of exact gradients, you estimate them from how the loss changes when you perturb the parameters slightly. This estimation is noisy, and for years that noise made zero-order methods impractical. MeZO, a representative zero-order method, required 100,000 steps to reach 91.4% accuracy on SST-2 with OPT-13B. Backpropagation did better and finished sooner. The Stanford team’s question is whether zero-order convergence is fundamentally limited or limited by avoidable design choices about compute budget allocation.
How 1.5-SPSA Works
The foundation is 1SPSA, a standard zero-order gradient estimator. At each training step, you draw n_pert independent perturbation directions z_i, where each z_i is a Rademacher vector with entries in {-1, +1}. You evaluate the loss at theta plus epsilon times z_i and at theta minus epsilon times z_i, take the difference, and use that scalar to estimate the gradient contribution from that direction. Averaged across all n_pert perturbations, the estimator is:
Δ(theta) = (1 / 2 * n_pert) * sum over i of [L(theta + epsilon * z_i) – L(theta – epsilon * z_i)] / (2 * epsilon) * z_i inverse
The relative gradient variance of this estimator scales as M / n_pert, where M is the number of jointly perturbed parameters. More perturbations reduce variance; more parameters inflate it. For a fixed compute budget, you can spend it on many steps with few perturbations per step or fewer steps with many perturbations per step. Standard implementations have defaulted to many steps. That turns out to be the wrong choice.
The Budget Reallocation Insight
The first key finding in the 1.5-SPSA paper is about how to spend a fixed forward-pass budget. Rather than many small steps, reallocating toward large effective batch sizes with many perturbations per step dramatically improves accuracy. At effective batch size 128 and n_pert 160, 1SPSA reaches 94.2% accuracy on SST-2 in 80 steps using 205,000 forward passes total. MeZO’s 91.4% required 100,000 steps but far more compute time. The number of optimizer steps is not the bottleneck; gradient quality per step is.
What’s easy to miss is the memory implication. Running large effective batch sizes in inference mode keeps memory at 60 GB regardless of batch size, because each forward pass overwrites the previous one. In training mode, memory grows with batch size due to stored activations. That asymmetry is what makes budget reallocation practically useful rather than trivially obvious.
Probe-Space Preconditioning
1.5-SPSA extends 1SPSA with a single additional forward pass per step, an unperturbed “clean” evaluation of the loss. That evaluation costs relatively little but enables something that standard zero-order methods cannot do cheaply: estimating curvature in the probe space.
The idea is to estimate a diagonal preconditioner in the n_pert-dimensional probe space rather than the full d-dimensional parameter space (which would require O(d) storage and defeat the purpose). For each perturbation direction i, the method estimates a curvature weight w_i = |c_hat_i| raised to the power alpha = 0.1, using the difference between the clean loss and the perturbed loss along that direction. High curvature means small steps are unstable, so high w_i directions get down-weighted. This is what Adam does approximately in parameter-space, done here in probe-space for a fraction of the cost.
The gain is proportional to the Hessian condition number kappa. In high-kappa landscapes, 1.5-SPSA converges up to 6x faster than 1SPSA. You don’t need to know kappa in advance; you get the benefit whenever it’s high. When kappa is near 1, 1.5-SPSA and 1SPSA are nearly identical, so there’s no real downside to adding the preconditioner.
Implementation matters here. The Stanford team built 8-bit-packed random generators and Triton fused unpack-and-apply kernels to keep perturbation generation from dominating step time, and added distributed parallelism for in-place OPT-30B training on commodity A100 GPUs. Splitting perturbation evaluation across devices requires no gradient communication because there are no gradients to communicate. These engineering choices are what make the results transfer from benchmarks to real hardware.
SOMA: Scaling Zero-Order to Pretraining
The second Stanford paper introduces SOMA (Sharded Optimization Mixture of Assemblies), published the same day at arXiv:2609.37899. SOMA addresses a different version of the variance problem. Recall that 1SPSA’s gradient variance scales as M / n_pert, where M is the total number of perturbed parameters. For a large model, M is large by definition, and you cannot reduce it without reducing model capacity. SOMA reduces variance by restructuring the model architecture so that the relevant M per gradient estimate is not the total model size but the size of a single expert.
SOMA uses LSTM experts rather than transformers, deliberately: each transformer requires a growing KV cache at inference time, which would undermine the memory advantage that zero-order training provides. The architecture uses a shared, weight-tied embedding and decoder matrix E, two residual blocks per expert (one LSTM, one MLP), and a fixed router that selects top-k experts per input using tf-idf weights reduced to 128 dimensions by SVD. Expert predictions are averaged.
The training process has five stages: train an N=1 seed for 1,000 updates on a 10-billion-byte subset of FineWeb-Edu; cluster the 100-billion-byte corpus with word-level tf-idf, 128-dimensional SVD, and balanced spherical k-means; copy the seed body to N experts; train each expert independently with SPSA on its data cluster (one expert per GPU, no gradient or activation exchange); then at inference, route tokens to top-k experts using tf-idf weights and average predictions.
The separable loss function is the mechanism that makes this work. Because each expert trains on its own data cluster, perturbations applied to one expert introduce no noise into another expert’s gradient estimate. The ensemble’s gradient variance scales with expert size rather than total ensemble size, which is the opposite of what happens when you scale a monolithic zero-order model. The measured relative centered variance for SOMA N=256 is 118 times lower than for an N=1 control.
Results
The 1.5-SPSA results are concrete. On SST-2, 1.5-SPSA achieves accuracy 3.1 percentage points higher than both MeZO and backpropagation in 70 steps. For OPT-13B specifically, 1.5-SPSA reaches 94.5% in 70 steps using 179,000 forward passes total. MeZO required 100,000 steps and reached only 91.4%. Testing on Qwen3-1.7B and Qwen3-8B across 6 post-training datasets shows consistent improvements over both 1SPSA and backpropagation. The preconditioner also expands the stable learning-rate range, reducing hyperparameter sensitivity that previously made zero-order methods brittle in practice.
The SOMA results require more caution. The experiments use 8.44M parameter models. That is intentionally small: the paper is studying zero-order scaling behavior, not demonstrating production-ready pretraining. At N=2, SOMA reaches 1.76 test nats/byte versus 2.00 to 2.11 for monolithic SPSA baselines and 2.21 for EGGROLL at roughly 150 aggregate GPU-hours. SOMA N=256 achieves 2.36 million tokens per second versus 257,000 for SOMA N=8, a 9.19 times speedup including routing overhead. The compute cost for N=256 is 59.9 times higher aggregate training compute than N=8, and the team used approximately 80,000 RTX 5090 GPU-hours to study SOMA configurations, which points to why scaling these results to GPT-3-scale parameters remains an open problem.
Limitations and Open Questions
The 1.5-SPSA results are for fine-tuning, not pretraining. Fine-tuning from a good pretrained checkpoint starts in a region of parameter space where the loss landscape is already relatively well-conditioned. The paper does not claim that 1.5-SPSA can replace backpropagation for pretraining from random initialization. The benchmark is SST-2, a binary sentiment classification task. SST-2 results do not automatically generalize to tasks like multi-step reasoning or code generation, where the loss landscape structure is quite different.
The SOMA architecture uses LSTMs rather than transformers. LSTMs have known limitations on tasks requiring long-range dependencies. The routing mechanism uses tf-idf features rather than learned representations, so the router cannot adapt to distributional shift. The separability that makes SOMA’s variance reduction work also means experts never share representations, which wastes capacity on overlapping domains. And the SOMA experiments use 8.44M parameter models only; scaling to GPT-3-class parameters remains an open problem. The N=256 configuration uses 59.9 times more aggregate training compute than N=8, which is a steep price for a method still at research scale.
The condition number kappa governs how much 1.5-SPSA improves over 1SPSA. High kappa means large gains. But kappa is not directly observable without computing the Hessian, which requires backpropagation. The Qwen3 and OPT results suggest the preconditioner generally helps, but the magnitude varies across tasks and the paper does not provide a rule for predicting when the benefit will be small enough to skip the extra forward pass.
What This Means for Engineering Teams
The immediate practical consequence of 1.5-SPSA is that fine-tuning large models no longer requires provisioning infrastructure sized for backward passes. If you are currently unable to fine-tune OPT-30B-class models because 600 GB of GPU memory is outside your infrastructure budget, 60 GB is a fundamentally different conversation. A single 8-GPU machine with A100 80GB cards could potentially handle OPT-30B fine-tuning in inference mode where Adam-based training would require a multi-node cluster with expensive interconnects.
The budget reallocation result also has direct implications for teams already doing zero-order optimization and getting poor results. If your current setup runs many steps with few perturbations per step, reallocating toward fewer steps with larger effective batch sizes and more perturbations per step may improve accuracy without changing the total forward-pass count. That is a configuration change, not an algorithmic one. Teams exploring PyTorch training optimization techniques will find the budget reallocation analysis directly applicable to their existing zero-order setups.
For teams building or deploying large models where memory is the limiting factor, fine-tuning at inference-mode memory cost means iterating on more model variants within the same hardware budget, running longer context lengths, or reaching models that were previously inaccessible without cluster-level infrastructure. Teams evaluating AI model selection and deployment for production use cases should treat zero-order fine-tuning as a viable option when backpropagation infrastructure is cost-prohibitive. Teams working on AI engineering services that need to scale fine-tuning across many model variants will find the 10x memory reduction the most immediately compelling number in this paper.
Key Takeaways
- 1.5-SPSA achieves 3.1 percentage points higher accuracy on SST-2 than both MeZO and backpropagation in 70 steps, while training OPT-30B in approximately 60 GB of GPU memory versus 600 GB for Adam-based training, a 10x memory reduction.
- The core mechanism is two independent improvements over standard zero-order training: reallocating compute budget from many steps toward fewer steps with large effective batch sizes and many perturbations per step, and adding a single clean forward pass per step to estimate a diagonal preconditioner in probe-space that down-weights high-curvature directions.
- Probe-space preconditioning yields gains proportional to the Hessian condition number kappa: in high-kappa loss landscapes, 1.5-SPSA converges up to 6x faster than 1SPSA, and the preconditioner expands the stable learning-rate range across Qwen3-1.7B, Qwen3-8B, and OPT model families.
- SOMA reduces gradient variance 118x for N=256 versus N=1 by sharding training into independent LSTM experts on clustered data, where variance scales with expert size rather than total ensemble size, enabling ensemble growth without degrading per-expert gradient estimates.
- SOMA’s results are at 8.44M parameters only, N=256 training uses 59.9x more aggregate compute than N=8, and the LSTM architecture has not been tested at transformer scale, making SOMA a research result rather than a production-ready pretraining recipe.
- All training and evaluation code and model checkpoints are publicly released, making both 1.5-SPSA and SOMA directly adoptable for teams with the hardware budget to run inference-mode OPT or Qwen3 fine-tuning experiments.
Work With Origins AI
Origins AI builds production AI systems for engineering teams. If you need to fine-tune large models without the GPU memory overhead of full backpropagation infrastructure, or want to evaluate whether zero-order optimization is practical for your specific fine-tuning tasks and hardware budget, talk to our team.