The MoE Era: Models With 397B Parameters That Only 'Run' About 17B

In 2026, open-weight frontier models closed the gap with proprietary systems faster than expected. Behind that leap is a single architectural idea: Mixture-of-Experts (MoE).

What MoE does

In a classic 'dense' model, every token passes through all parameters. In MoE, the model is split into many 'expert' sub-networks and a router activates only a few experts per token. The result: total capacity stays enormous while the compute done at each step is far smaller.

2026's open MoE models

• Alibaba Qwen3.5-397B-A17B: 397B total parameters, ~17B active per token; multimodal reasoning and ultra-long context.
• Z.ai GLM-5.3-Flash: 320B/18B sizing, 1M context, MIT-licensed weights; natively multimodal.
• Kimi K2.5: a multimodal agentic model built via continual pretraining on ~15T mixed visual + text tokens.

Why it matters: cost and deployment

MoE pairs 'frontier quality' with 'reasonable inference cost'; models that are huge yet downloadable and deployable are now within reach of any team with adequate hardware. But the active/idle parameter balance, routing quality and memory footprint require careful management in production.

The remot angle

remot lets you use dense and MoE models side by side in one orchestration layer: it routes work to the right model (open or proprietary), measures latency and cost, and holds a quality bar. That way architectural diversity becomes an optimizable pool of options rather than chaos.