Mixtral and Mixture-of-Experts, Explained

Mixtral 8x7B carries roughly 47 billion parameters but only fires about 13 billion of them for any given token, and it still matches or beats Llama 2 70B and GPT-3.5 on most benchmarks the authors test. That is the whole pitch of Mixtral of Experts: get the quality of a large model while paying the inference bill of a small one, by routing each token through only a fraction of the network.

TL;DR

Mistral AI took their Mistral 7B architecture and replaced each layer's single feed-forward block with eight expert feed-forward blocks plus a router that picks two of them per token. Total capacity goes up to ~47B parameters; active compute per token stays near ~13B. The result matches or surpasses Llama 2 70B and GPT-3.5 on most metrics — clearly ahead on mathematics, code generation, and multilingual benchmarks, narrowly behind on a few commonsense and reading-comprehension ones — and shipped openly under Apache 2.0, which is largely why sparse mixture-of-experts went from a research curiosity to a default tool in the open-weight community.

The problem

Dense transformers get better as you scale them, but the cost structure is unforgiving: at inference every parameter participates in processing every token. Double the parameters and you roughly double the per-token compute. There is no way to hold a lot of knowledge in the weights without paying for all of it on every forward pass.

The question Mixtral asks is whether total capacity and per-token compute can be decoupled. You want a big pool of parameters available — the model should know a lot — but you do not want to activate the entire pool to predict the next token, most of which has nothing to do with whatever specialized knowledge a given token needs. Sparse mixture-of-experts is the mechanism that breaks that coupling.

How it works

The backbone is close to Mistral 7B, but the paper names two exceptions, not one: Mixtral "supports a fully dense context length of 32k tokens," and the feed-forward blocks are replaced by MoE layers. Attention is still a single shared block per layer — it is not replicated across experts — which is what keeps the total parameter count well below 8×7B. Concretely, each layer holds eight expert FFNs and a small router (a gating network) that decides which experts handle each token.

For each token at each layer, the router scores all eight experts, keeps the top two, and discards the rest. Concretely, the gate computes G(x) = Softmax(TopK(x · W_g)), where every expert outside the top-k has its logit set to −∞ before the softmax, so it contributes exactly zero. The layer's output is the softmax-weighted sum over the selected experts:

y = Σ_i G(x)_i · Expert_i(x)

Two details matter here. Routing is per token and per layer — the choice of experts is made independently at every layer and every position, so the same token can take different experts as it moves up the stack, and adjacent tokens can diverge. And because only two of eight experts run, the active parameter count stays low even though all eight sets of weights exist.

The arithmetic is the headline. The paper reports the total at about 47B parameters without deriving the figure; the arithmetic lands below 8×7B because attention and embeddings are shared rather than duplicated across experts. Only two experts run per token, keeping active parameters near 13B — roughly 5× fewer active parameters than Llama 2 70B. That is why Mixtral carries the compute cost of a ~13B model while holding ~47B worth of capacity. It does not reach 13B-dense throughput in practice: the paper notes the SMoE layer "introduces additional overhead due to the routing mechanism and due to the increased memory loads when running more than one expert per device," and that such layers "are more suitable for batched workloads."

Beyond the routing, Mixtral is trained with a 32k-token context window and is multilingual across English, French, Italian, German, and Spanish. There is also Mixtral 8x7B – Instruct, a chat variant produced by supervised fine-tuning followed by Direct Preference Optimization (DPO).

Results

Against Llama 2 70B, Mixtral wins most of the table, with the paper noting one stated exception: Figure 3's caption says Mixtral outperforms Llama 2 70B on all benchmarks "except on reading comprehension benchmarks." On commonsense suites the two trade places — Llama 2 70B is still ahead on HellaSwag (85.4 vs 84.4) and WinoGrande (80.4 vs 77.2), while Mixtral takes ARC-Challenge (59.7 vs 56.5). The gap opens decisively on math and code.

All figures from Table 1 (GSM8K 8-shot maj@8). The paper's Table 2 reports GSM-8K at 5-shot: 58.4% vs 53.6%.

Against GPT-3.5, the paper reports Mixtral performing "similarly or above" on most metrics (MMLU 70.6 vs 70.0, MBPP 60.7 vs 52.2, GSM-8K 58.4 vs 57.1 at 5-shot), while GPT-3.5 stays narrowly ahead on WinoGrande (81.6 vs 81.2) and MT-Bench (8.32 vs 8.30).

The instruct model holds up against closed chat models of its era: Mixtral 8x7B – Instruct reaches MT-Bench ≈ 8.30, essentially level with GPT-3.5 Turbo (≈ 8.32) and the strongest open-weight chat model at release. In human and Arena-style preference evaluation the paper reports it surpassing GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B-chat. The authors also report less measured bias and a more positive sentiment profile than Llama 2 — though the paper measures bias on the base model (§3.3, BBQ 56.0% vs Llama 2 70B 51.5%), while its introduction attributes the same property to the Instruct model.

What the experts actually learn

The most interesting section of the paper is not the leaderboard — it is the routing analysis, because the result is counterintuitive. The paper's own framing is modest, calling this "a small analysis," but the finding is the one people remember: you might expect a "biology expert" and a "code expert" to emerge, and they do not.

When the authors inspect which experts handle text on biology (PubMed Abstracts), philosophy (PhilPapers), or LaTeX-heavy ArXiv papers, they find little evidence of topical specialization — the paper reports that "the distribution of expert assignment is very similar" across these domains at every layer inspected. The one exception the paper flags is DM Mathematics, where it notes "a marginally different distribution of experts" — which the authors attribute to that subset's synthetic nature and narrow coverage of natural language rather than to genuine domain expertise.

What the router does key on is structure and syntax: certain experts are consistently favored for particular token types, such as indentation and keywords in code. There is also positional locality, but it is layer-dependent. The paper reports that "repetitions at the first layer are close to random, but are significantly higher at layers 15 and 31" — first-choice repetition sits near the 12.5% random baseline at layer 0, rising to roughly 24–28% at layers 15 and 31.

The takeaway is that "expert" is a misleading word. These are not human-meaningful, domain-aligned modules. Expert assignment appears to align more with syntax than with domain, especially at the first and last layers — low-level structural regularity, not high-level semantics.

Limitations and context

The efficiency story has an asterisk worth saying plainly. Sparsity cuts the FLOPs per token, but it does nothing for memory: all 47B parameters must sit in VRAM, because the router might select any expert at any layer. So Mixtral is cheap to run but expensive to host — it has the compute profile of a 13B model and the memory footprint of a 47B one, which the paper points out is still smaller than Llama 2 70B's. If you are GPU-memory-bound rather than compute-bound, the win shrinks.

Serving is also genuinely harder. Top-2 routing creates load-balancing and batching problems — different tokens in a batch want different experts — so efficient deployment needs expert-parallelism and carefully written kernels rather than a naive forward pass.

A few other caveats: the lack of semantic specialization means you cannot point at an expert and reason about or control it, which complicates interpretability. The paper is also light on pretraining-data details and the exact training recipe, so full reproduction is not on the table. And several reported scores depend on shot counts and maj@k decoding, so cross-paper comparisons should match settings before drawing conclusions.

Why it matters

Mixtral was a watershed open-weight release. It demonstrated that sparse MoE could deliver frontier-class quality at a fraction of the active compute, and it did so under a permissive Apache 2.0 license that let anyone build on it. That combination seeded a wave of open MoE models and a corresponding wave of serving optimizations, and it made "active parameters versus total parameters" a standard axis for reasoning about model efficiency. The routing analysis, meanwhile, opened a durable research thread on how — and whether — experts specialize at all, a question the field is still working through.

References

• Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., et al. (Mistral AI). Mixtral of Experts. arXiv:2401.04088, 2024. https://arxiv.org/abs/2401.04088
• HTML version: https://ar5iv.labs.arxiv.org/html/2401.04088
• Mistral AI announcement: https://mistral.ai/news/mixtral-of-experts/