One Model, Multiple Compute Budgets:
Multi-Ratio ConceptMoE
for Flexible Compute Across Turns

One shared model  ·  CR2 / CR4 / CR8  ·  flexible compute across turns

Yiyang Fu1,2,*, Zihao Huang2, Dongchen Han2,3,*, Shukai Gong1,
Zhen Dong4, Qiyang Min2,†, Daquan Zhou1,†
1 Peking University2 ByteDance Seed3 Tsinghua University4 University of California, Santa Barbara
* Work done at ByteDance Seed · † Corresponding authors
Paper Demo code BibTeX
Flexible compute within a shared model. Multi-Ratio ConceptMoE changes compression ratios across turns and improves the accuracy-compute trade-off on CodeChat and CoSQL.
2 / 4 / 8
Compression ratios
in one model
6 of 6
Wins against
compute-matched MoEs
78.4%
Fewer FLOPs than model routing

CodeChat · same turn-wise schedule

Abstract

Compute demand changes across conversational turns. Using a large model throughout can waste computation, while switching between independent models may require rebuilding their history states.

We introduce Multi-Ratio ConceptMoE, which jointly supports compression ratios of 2, 4, and 8 within a shared backbone. Lightweight full-token modules reduce compression-independent cost, ratio-specific boundary routers expose distinct compute budgets, and cross-ratio training prepares the model to process different compression granularities in the same context.

Across PTBench and SFTBench, all three fixed modes outperform their compute-matched MoE baselines. On CodeChat and CoSQL, mixed-ratio inference improves the accuracy-compute trade-off while avoiding the history re-prefill overhead accounted for when routing among independent models.

Method: Multi-Ratio ConceptMoE

Only the boundary routers are ratio-specific. The encoder, concept backbone, DeChunk module, and decoder are shared.

The Multi-Ratio ConceptMoE architecture. Each input is encoded once. Ratio assignments select candidate boundaries; segment starts force a fresh concept. Compression views are packed for shared concept-level processing in the full model.

01Encode onceEvery router reads the same token representations.

02Select boundariesFixed or mixed assignments define each compression view.

03Share computationThe same backbone and decoder process all ratios.

EXPLORE THE COMPRESSION MODES

One conversation, one model

Illustrative schematic

Fixed: use one ratio for every turn. Mixed: change the ratio between turns, using the same model.
CR is the number of input tokens per concept: a larger CR produces fewer concepts for the backbone to process.

Mixed: Turn 1 → CR2, Turn 2 → CR8, Turn 3 → CR4. Different turns use different ratios within one conversation.

TURN 01CR2

Solve the initial task

Write a complete implementation.

24 new tokens (example)
24 ÷ 2 = 12 concepts
TURN 02CR8

Make a small refinement

Rename variables and tidy formatting.

24 new tokens (example)
24 ÷ 8 = 3 concepts
TURN 03CR4

Handle the edge cases

Add validation and boundary handling.

24 new tokens (example)
24 ÷ 4 = 6 concepts

One growing concept sequence

After three turns: 12 + 3 + 6 = 21 concepts in one sequence.

T1 → T2 → T3 are successive turns of the same conversation. Each square represents one concept.

One shared concept backbone Same weights for every ratio

Each new turn appends concepts at its chosen ratio. Earlier concepts and their history states stay unchanged.

This panel illustrates turn-wise inference with an example ratio schedule. Cross-ratio training instead samples ratios for contiguous segments within a sequence. Equal-sized token groups simplify the illustration; actual boundaries are learned and concept counts vary.

Experimental Results

Strong at every fixed budget

6 wins in 6 comparisons

CR2, CR4, and CR8 outperform compute-matched Large, Medium, and Small MoEs on both evaluation suites.

Accuracy of fixed compression modes and their compute-matched MoE baselines
Compute budgetModePTBench MoE → OursSFTBench MoE → Ours
HighCR231.7→32.119.0→21.5
MediumCR428.2→29.315.5→19.1
LowCR825.4→27.613.1→17.2

Accuracy (%). Compute matching excludes sequence-length-dependent attention-map computation, following the paper's setup.

MIXED-RATIO INFERENCE

More accuracy. Less compute.

Combined Joint accuracy43.78%

+5.87 pp over the Large MoE

Realized computation1.0999 PFLOPs

41.1% fewer FLOPs than the Large MoE

CodeChat · terminal-turn workload
ModelAccuracy (%) ↑FLOPs (PF) ↓
MoE Large37.911.8662
MoE Medium27.091.1198
MoE Small18.680.7334
Multi-Ratio CR243.281.3072
Multi-Ratio CR435.370.9529
Multi-Ratio CR829.290.7795
Multi-Ratio Mixed Ours43.781.0999
SingleCR2 specialist33.892.2290
SingleCR4 specialist24.601.2784
SingleCR8 specialist18.020.7326

Results from the trained models in arXiv:2610.14074v1. CodeChat uses Combined Joint accuracy; CoSQL uses SQLite execution accuracy. Multi-turn FLOPs include attention-map computation under the terminal-turn workload protocol.

Switching Compute Budgets

Independent models maintain incompatible history states. Switching can add costly history re-prefill as a conversation grows.

Same schedule, different switching cost. The routed baseline includes the extra history re-prefill needed after each model switch. This comparison measures algorithmic FLOPs, not wall-clock latency.
CodeChat5.0895 → 1.0999 PFLOPs
CoSQL0.6250 → 0.3319 PFLOPs

Independent-model routing → Multi-Ratio Mixed
Using the same frozen turn-wise budget schedule.

Cross-Ratio Training

Training on cross-ratio patterns strengthens mixed-ratio inference while largely preserving fixed-ratio capability.

Cross-ratio exposure matters. Relative to three homogeneous training views, three cross-ratio views improve mixed inference by 6.14 pp on CodeChat and 8.84 pp on CoSQL.

Minimal Demo

Independent routers. Shared modules. Fixed-ratio and mixed-ratio views, following the original ConceptMoE demo structure.

Download demo
MultiConceptMoE_demo.pyPython
# Install dependencies and run
python3 -m pip install torch einops
python3 MultiConceptMoE_demo.py

# Inside the demo: one view per ratio
logits, aux_loss = model(input_ids)

# Independently sampled mixed-ratio views
logits, aux_loss = model(
    input_ids, mixed_cr=True
)

The demo uses random weights and placeholder attention/MoE layers to explain routing. It does not reproduce trained-model results or implement inference caches.

BibTeX

@misc{fu2026multiratio,
  title = {One Model, Multiple Compute Budgets:
           Multi-Ratio ConceptMoE for Flexible Compute Across Turns},
  author = {Fu, Yiyang and Huang, Zihao and Han, Dongchen and
            Gong, Shukai and Dong, Zhen and Min, Qiyang and Zhou, Daquan},
  year = {2026},
  eprint = {2610.14074},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2610.14074}
}

↑
Paper figure