Six Small Models That Schedule the Compute of Bigger Models

A trained Qwen2.5-7B model now steers a frozen Qwen2.5-32B answer generator well enough to outperform the concepts the 32B produces for itself, and transfers to a Llama-3.3-70B from a different model family it was never trained against. That result, from Ismail Labiad et al. at Meta FAIR (arXiv:2609.26704, September 22, 2026), crystallises something that has been building across multiple research groups through 2026: the hard part of inference-time compute scaling is not deciding how much to spend, it is deciding where to spend it. The systems covered below are not refinements on “add more sampling.” They are separate, reusable components that sit above a frozen target model and make allocation decisions the target model cannot make for itself.

Why This Comparison Is Timely

The dominant test-time compute playbook through 2025 was simple: draw N independent samples from your model, vote or verify, keep the best. As a compute strategy it works, but it scales poorly. Every sample is statistically independent, meaning the second sample has the same probability of recovering a genuinely new idea as the first. On hard problems where the first 10 samples are all wrong, the next 10 rarely fix the issue. Meta FAIR’s “Beyond Repeated Sampling” (September 22, 2026) quantifies this directly: naive repeated sampling with identical decoding parameters yields near-duplicate reasoning paths, and the authors demonstrate that the apparent gains of earlier concept-guided methods vanish entirely once the repeated-sampling baseline is allowed to use exploratory decoding parameters (temperature 1.0, top-p 0.95).

What follows from that finding is the framing this comparison examines. If naive sampling is structurally inefficient, the right next question is not “how many samples” but “which controller decides.” Four groups independently published learned allocation policies between April and August 2026, each targeting a different resource: token sampling budgets, per-step rollout counts, denoising schedule steps, or exploration direction. A fifth paper from Carnegie Mellon (June 30, 2026) extends the idea to robotics. Together they define a class of runtime components that did not exist two years ago: a learned inference scheduler that wraps a frozen target model and allocates its compute dynamically per query or per decision step.

The Shift: Fixed Decode Procedure to Learned Inference Policy

The old picture is: your application calls a model; the model runs a fixed sampling procedure; you get output. The procedure (greedy, top-p sampling, best-of-N) is a hyperparameter chosen at deployment time and applied uniformly to every input. This worked because model capability was the bottleneck. When the model was right, one sample was enough. When it was wrong, no number of identical samples helped.

The new picture inserts a learned policy between the application and the model invocation: “application calls a controller; the controller inspects the query and emits a compute schedule; the model executes that schedule and returns output.” The controller can be an RL-trained concept generator, a Lagrangian-relaxation classifier, an agreement-monitoring agent, a fuzzy rule system, a latent novelty estimator, or a meta-MDP policy trained over denoising iterations. All of them share one property: the target model’s weights are frozen, and the controller modifies only what the model is asked to do, not what it knows. That separation is what makes these controllers reusable across model families.

Beyond Repeated Sampling: RL-Trained Concept Generator as Search Policy

Labiad, Kowalski, Schoenauer, Munos, and Kempe (Meta FAIR and Universite Paris-Saclay, arXiv:2609.26704, September 22, 2026) train a 7B parameter concept generator with reinforcement learning to produce problem-specific reasoning directions for a frozen 32B answer generator. The mechanism works in two stages: the concept generator emits a full batch of concepts in a single autoregressive trajectory (not one concept at a time as earlier work required), and the answer generator produces responses conditioned on each concept. The concept generator is trained with trajectory-level RL rewards tied directly to whether the answer generator succeeds on a given problem, so the generator learns to produce concepts that are useful to that specific downstream model rather than concepts that look superficially relevant.

The numbers are direct. On hard held-out DeepMath problems where naive repeated sampling achieves a pass@128 of 19.0%, the trained concept generator raises this to 39.2% at identical answer-generation compute allocation. The generator was trained against a Qwen2.5-32B answer generator, but transfers without retraining to Llama-3.3-70B, a model from a different family. The trained 7B concept generator also outperforms an untuned Qwen2.5-32B generating its own concepts. This last point is important: the controller is not merely distilling capability from a larger untuned model; it is learning something the larger model does not know how to do for itself, namely schedule exploration at the semantic level rather than the token level.

The single-trajectory design matters for deployment. Earlier iterative concept generation procedures break continuous batching and KV-cache sharing in serving engines like vLLM because they alternate between two models in a loop. Generating all concepts in one pass from the concept generator and then running the answer generator in a standard batch preserves standard autoregressive throughput. The authors note this directly as an engineering motivation. The limitation: training the concept generator requires a verifiable reward signal (math problems where correctness can be checked automatically), and it is not yet clear how well the learned policy generalises to tasks where that signal is unavailable or noisy.

Adaptive Test-Time Compute via Constrained Policy Optimization: Provably Budget-Constrained Allocation

Zhiyuan Zhai, Bingcong Li, Bingnan Xiao, Ming Li, and Xin Wang (Fudan University and ETH Zurich, arXiv:2604.14853, April 16, 2026) formalise per-query compute allocation as a constrained optimisation problem: maximise expected accuracy subject to a global average compute budget constraint. The key insight is that Lagrangian relaxation decomposes this global constraint into independent per-instance sub-problems, each with a closed-form oracle solution. For a given dual variable (shadow price for compute), the oracle allocates the budget level that maximally balances marginal accuracy gain against compute cost. Because the aggregate cost under this oracle is monotone in the dual variable, binary search finds the exact dual variable that matches any target budget.

The second stage trains a lightweight classifier (a gradient-boosted machine) to predict oracle allocations from cheap input features that can be computed before running the large model. This “Solve-then-Learn” pipeline amortises the allocation rule: at deployment, only the cheap classifier runs, not the oracle. The authors prove a regret bound showing that the task-level accuracy gap between the learned policy and the oracle is bounded by the classifier’s imitation error times a worst-case per-instance gap. In practice, the classifier achieves over 91% imitation accuracy on held-out data.

Experiments run DeepSeek-V3, GPT-4o-mini, and Qwen2.5-7B on MATH and GSM8K. The method achieves up to 12.8% relative accuracy improvement on MATH at matched average budget constraints compared to uniform allocation baselines. The allocation pattern the system learns matches intuition: easy questions (trivially correct at one sample) consistently receive budget 1; “responsive” questions that exhibit steep accuracy gains with more samples receive high budgets; intractable questions that stay wrong regardless of compute receive fewer samples rather than more, preserving budget for the responsive subset. One limitation the paper acknowledges: the approach requires offline data collection to estimate accuracy as a function of budget for each input, which adds an upfront calibration cost before deployment.

TrACE: Agreement Between Concurrent Trajectories as a Free Difficulty Signal

Khushal Sethi (Stanford University, arXiv:2604.08369, April 9, 2026) takes the opposite methodological stance from the two entries above: no training, no reward model, no labelled data. TrACE (Trajectorial Adaptive Compute via Agreement) uses inter-rollout action agreement as a free signal of per-step difficulty in sequential agent tasks. At each decision step, the controller samples a small initial set of candidate next actions and measures how consistently the model commits to the same action across those samples. High agreement means the step is easy; the controller commits immediately. Low agreement means the step is hard; the controller samples additional rollouts up to a configurable cap before committing to the plurality action.

The agreement signal is behavioural consistency, not verbalized confidence. This distinction matters because LLMs are poorly calibrated when asked to report their own uncertainty in words, but their output distribution across parallel samples is a more reliable indicator of whether the model actually has a committed answer. The paper cites Xiong et al. (2024) to support this: behavioural consistency is more calibrated than verbalized confidence in chain-of-thought settings.

TrACE-4 matches SC-4 (fixed self-consistency at 4 samples) accuracy while using 33% fewer LLM calls on GSM8K (n=50) and 39% fewer on MiniHouse, a multi-step household navigation benchmark. TrACE-8 matches SC-8 while using 55% fewer LLM calls on GSM8K and 65% fewer on MiniHouse. All experiments use a Qwen 2.5 3B Instruct model running on CPU, which limits the size of the claim, but the per-step measurement approach is architecture-independent. The honest limitation from the paper: these results are at small scale (3B model, 30-50 evaluation instances per benchmark), and the method has not been compared against trained adaptive-compute methods on the same benchmarks. It is also unclear how sensitive the agreement threshold is across different tasks and models.

Interpretable Adaptive Sampling: Fuzzy Control Maps Signals to Budgets

Mobina Kashaniyan and Ali Jannesari (Iowa State University, arXiv:2608.03961, August 4, 2026) take an explicit interpretability-first position: instead of training a neural policy or using a heuristic threshold, they use a hierarchical fuzzy controller that maps human-readable signals to a per-query sampling budget. The controller inputs are estimated prompt complexity and model confidence, both normalised to [0, 1]. Complexity is computed from a weighted combination of surface features (prompt length, punctuation density, digit density, sentence count) and NLP-derived features (vocabulary richness, math-symbol density, semantic ambiguity, linguistic complexity, reasoning depth). Confidence is estimated from a short draft answer’s mean log-probability across generated tokens.

The fuzzy inference system uses triangular and trapezoidal membership functions for low, medium, and high regions on each input dimension. A rule base maps combinations of complexity and confidence levels to a continuous budget scale value, which is then converted to an integer sample count. Because each rule is human-readable (“if complexity is high and confidence is low, then budget is high”), the allocation decision for any prompt can be fully audited, a property that learned neural policies do not offer. The controller adds a history cache that tracks slow-moving performance estimates on similar prompts indexed by coarse complexity and prompt-type bins, giving the system lightweight online adaptation without retraining.

Across question-answering and mathematical reasoning benchmarks, the fuzzy controller matches or improves over fixed-budget baselines including best-of-N and self-certainty-based methods, while reducing average sample count. The paper evaluates under a fair-alignment protocol where decoding parameters are matched and the answer selector is held constant, so changes in accuracy are attributable only to the budget policy. The key limitation the authors acknowledge: the fuzzy membership functions and rule bases were manually designed, meaning the system’s behaviour reflects the designer’s prior beliefs about what signals matter. A misconfigured rule base could systematically over- or under-allocate in ways that are visible in the audit trail but not automatically corrected.

Latent Distilling: Novelty in Hidden-State Transitions Steers Exploration

Yuanhao Zeng, Ao Lu, Lufei Li, Zheng Zhang, Yexin Li, and Kan Ren (BIGAI and ShanghaiTech University, arXiv:2604.24927, April 27, 2026) approach the diversity problem from inside the generation process rather than above it. Their method, Exploratory Sampling (ESamp), trains a lightweight Latent Distiller at test time to predict deep-layer hidden representations of the LLM from its shallow-layer representations. The distiller adapts online across the candidate batch: familiar representation mappings produce low prediction error; unfamiliar mappings produce high error. High prediction error signals that a candidate token extension would take the model into a less-explored semantic region.

This novelty signal is incorporated into decoding through a KL-regularised optimisation objective. The closed-form optimal solution reweights the base model’s next-token distribution by an exponential function of the novelty reward, suppressing probability mass on continuations that correspond to familiar representation mappings. Because the distiller updates online across all parallel sequences in a batch, the trajectories implicitly coordinate: a reasoning pattern explored by one sequence in the batch is penalised in subsequent sequences, encouraging the batch as a whole to cover more of the semantic space. This coordination happens at the representation level rather than at the token level, which is what distinguishes ESamp from methods like contrastive decoding that suppress token repetition directly.

The asynchronous pipeline decouples distiller training from LLM generation, incurring less than 5% throughput overhead in standard serving scenarios (1.2% in the optimised release). ESamp significantly boosts Pass@k efficiency on AIME 2024, AIME 2025, LiveCodeBench, and GPQA. The paper also reports that ESamp breaks the trade-off between diversity and coherence in creative writing benchmarks where other diversity-promoting methods degrade output quality. The open-source implementation is released as the tLLM framework at https://github.com/LinesHogan/tLLM. The limitation: the distiller is a new model component that must be initialised and updated at test time, and its effectiveness depends on how predictive shallow-to-deep representation mappings are for the specific model architecture. The paper validates this for standard transformer architectures but notes the approach may need adjustment for architectures with significantly different depth profiles.

ELASTIC: Meta-Policy Allocates Denoising Steps and Parallel Samples for Robot Control

Andrew Zou Li, Gokul Swamy, Yonatan Bisk, and Andrea Bajcsy (Carnegie Mellon University, arXiv:2606.31132, June 30, 2026) extend the controller-over-frozen-model pattern to robotics. Generative control policies (diffusion policies, flow-matching models, vision-language-action models) expose two axes of test-time compute: sequential scaling (more denoising steps for higher precision) and parallel scaling (more independent action samples to cover multiple modes of the action distribution). The optimal allocation along these two axes depends on the task, the current robot state, and the base policy’s competence, all of which are unknown in advance.

ELASTIC formulates compute allocation as a meta-Markov Decision Process in which a meta-policy observes the robot state and selects the number of denoising steps and the number of parallel samples at each denoising iteration. The meta-policy is trained with RL against the frozen base policy, without access to that policy’s training data. The reward balances task success against compute cost. At each denoising iteration the meta-policy can decide: refine the current samples with more denoising steps, or abandon low-quality samples and draw fresh ones, or commit and execute.

In simulated manipulation benchmarks with diffusion policies, ELASTIC Pareto-dominates fixed and single-axis scaling baselines at matched compute budgets. On real hardware with the pi0.5 vision-language-action model, ELASTIC matches Best-of-10 success while reducing wall-clock latency by 34%. The latency reduction is particularly notable in robotics, where inference time directly translates to control delay. The paper notes that the RoboMonkey baseline (a parallel-only approach with 16 samples) incurs over 4 times the base policy’s inference latency even with a specialised serving engine. ELASTIC achieves comparable success at a fraction of that overhead by mixing sequential and parallel compute dynamically. The limitation: the meta-policy must be trained separately for each base policy and environment, which adds an RL training step before deployment and may require re-training when the base policy changes.

How They Compare

What This Category Reveals

The entries above are ordered from broadest architectural scope to most domain-specific. Beyond Repeated Sampling proposes a reusable cross-family controller trained with RL and validated on cross-family transfer. Adaptive TTC via CPO provides the most theoretically grounded framework, formalising the problem as a constrained optimisation and proving regret bounds. TrACE is the most practical to deploy immediately, requiring no training at all. Interpretable Adaptive Sampling prioritises auditability over raw performance. Latent Distilling operates at the most granular level, intervening at each token step rather than at the query level. ELASTIC extends the pattern to a completely different modality and exposes the two-axis allocation problem that is unique to diffusion-based generative control.

The ordering rationale is: widest generalisation first, then theoretical rigour, then deployment simplicity, then interpretability, then granularity of control, then domain extension. No single system dominates across all dimensions, which means the right choice for a team depends on what they need most. If you can train and want cross-family reuse, Beyond Repeated Sampling. If you have a verifiable budget constraint and want formal guarantees, Adaptive TTC. If you need to ship something today with no training data, TrACE. If you need to audit every allocation decision, the fuzzy controller. If you want to encourage semantic diversity at the token level with minimal overhead, ESamp. If you’re deploying diffusion-based robot policies, ELASTIC.

What the category has not yet answered: none of these systems has been evaluated on heterogeneous production workloads where query difficulty distribution shifts continuously and where a miscalibrated controller fails in ways that are hard to detect without ground-truth labels. The benchmark coverage is predominantly mathematical reasoning (MATH, GSM8K, AIME) and constrained navigation (MiniHouse). Generalisation to open-ended tasks without automatic verifiers is still an open problem.

Limitations and Open Questions

The evidence base for this category is benchmark-heavy and relatively narrow. Mathematical reasoning provides clean verifiable signals that are ideal for training and evaluating controllers, but production workloads mix question types, difficulty levels, and user intents in ways that no current benchmark captures. A controller trained or calibrated on a math distribution may under-allocate for an out-of-distribution creative task or over-allocate for factual lookups, and neither failure produces an obvious error signal during deployment.

Miscalibrated controllers introduce a specific failure mode that does not exist in fixed-budget systems: the controller can save compute by doing too little useful work. With fixed budgets, a system that saves compute is either explicitly configured for lower accuracy or is buggy. With learned controllers, a policy that achieves low average compute consumption is also consistent with a policy that systematically skips samples that would have been productive. Distinguishing these cases requires held-out accuracy measurement, which adds evaluation overhead that many teams will skip.

Controller overhead is real even when small. ESamp reports less than 5% throughput overhead in standard serving scenarios, but 5% is not zero. TrACE’s initial sample set adds latency proportional to the initial batch size before any decision is made. The Adaptive TTC classifier runs cheaply, but the offline calibration step that generates oracle labels requires running the full model at multiple budget levels on a representative dataset before the classifier can be trained. ELASTIC’s meta-policy requires a separate RL training run that must be repeated when the base policy changes.

The most fundamental open question is whether controllers trained on one task difficulty distribution maintain their efficiency advantage when that distribution shifts. Distribution shift in task difficulty is exactly the kind of covariate shift that causes under- or over-allocation in classifiers that have no online recalibration mechanism. TrACE and ESamp are immune to this in different ways (TrACE adapts per step with no stored model state; ESamp’s distiller adapts online). The learned-offline controllers (Adaptive TTC, Beyond Repeated Sampling) are more susceptible. None of the papers provides a systematic study of controller behaviour under deliberate distribution shift.

What This Means for Engineering Teams

For teams running reasoning models at scale, the practical question is not whether to allocate compute adaptively, but which of these controllers fits the deployment context. Teams with verifiable tasks and the capacity for offline calibration should look closely at Adaptive TTC: the formal guarantees and 91% imitation accuracy are attractive for production systems where budget constraints are contractual rather than aspirational. Teams deploying agents in multi-step sequential settings will find TrACE the fastest path from zero to adaptive allocation, at the cost of being unable to tune the controller’s behaviour beyond adjusting the agreement threshold.

The cross-family transfer property of Beyond Repeated Sampling is commercially significant. If a 7B controller trained against one target model transfers to a different target model without retraining, the controller becomes a persistent asset that outlives any individual model version. Teams running multiple model variants or planning model upgrades should evaluate this property carefully before committing to a controller that requires retraining when the base model changes.

The engineering team at Origins AI works on production AI inference stack optimisation across a range of model families and deployment environments. A useful framing from our work: the controller’s inference overhead should be measured as a fraction of the target model’s per-token cost, not as a fraction of total wall-clock time. At large target model sizes, controller overhead is negligible. At smaller target sizes (7B-13B) where latency is already tight, a 5% overhead is meaningful. Teams running efficient inference libraries where base throughput is already maximised will feel controller overhead more acutely than teams running standard serving stacks.

Two comparison dimensions matter most when evaluating these controllers for production. First, training cost and recalibration frequency: controllers that require offline oracle label generation (Adaptive TTC) or full RL training runs (Beyond Repeated Sampling, ELASTIC) have a periodic maintenance cost that scales with how frequently the target model or task distribution changes. Second, failure mode visibility: the fuzzy controller’s full rule audit trail gives operations teams a clear path to diagnosing allocation errors; RL-trained and learned policies require accuracy regression testing to surface the same failures.

Key Takeaways

• A 7B RL-trained concept generator raised a frozen 32B model’s pass@128 from 19.0% to 39.2% on hard math problems at identical answer-generation compute, and transferred without retraining to Llama-3.3-70B.
• Adaptive TTC via CPO achieved up to 12.8% relative accuracy improvement on MATH under matched budget constraints, with a lightweight classifier reaching over 91% imitation accuracy of the Lagrangian oracle.
• TrACE, requiring zero training, reduced LLM call counts by 33-65% across GSM8K and multi-step navigation benchmarks while matching fixed-budget accuracy.
• ESamp’s Latent Distiller adds less than 5% throughput overhead while improving Pass@k efficiency across mathematics, science, and code generation benchmarks, operating at the token level inside generation rather than above it.
• ELASTIC reduced wall-clock latency by 34% on real robot hardware with the pi0.5 VLA model while matching Best-of-10 task success, by jointly allocating denoising steps and parallel samples.
• None of these controllers has been evaluated under deliberate task-difficulty distribution shift, which is the most likely failure mode in production deployments.

Work With Origins AI

Origins AI builds production AI inference infrastructure for engineering teams. If your team is deciding how to allocate test-time compute across model families without rearchitecting every time a model version changes, talk to our team.