What "2.8 trillion parameters, 104 billion active" actually means

Moonshot AI has released the full weights for Kimi K3, and the headline number moving around the internet is 2.8 trillion parameters. That number is real, but it is not the number you should be reasoning with when you think about what running this model costs or how it compares to anything else. The number you want is 104 billion.

Here is what the gap between those two figures is, why it exists, and what it changes about how you should read this release.

The one-sentence version

Kimi K3 is a sparse mixture-of-experts model. Its brain contains 2.8 trillion parameters worth of specialists, but for any single word it produces, only about 104 billion of those parameters actually do any work. The rest sit quiet. According to Moonshot's own model card, reported by Digital Applied, K3 has 896 expert sub-networks, of which exactly 16 fire for each token.¹

The label on the box is the total parameter count. The invoice is the active parameter count. They are not the same number and they never were.

Why this design exists

A traditional "dense" model, think of the earliest GPTs or Llama, runs every parameter it has for every token it generates. If you have a 100-billion-parameter dense model, generating a token costs 100 billion parameters' worth of arithmetic. Doubling the model doubles the cost per token. That is a punishing scaling law for anyone paying an inference bill.

Mixture-of-experts (MoE) breaks that link. Instead of one giant network, you build many smaller specialist networks, the "experts", and put a tiny router in front of them. For each token, the router picks a handful of experts and sends the token only through those. The total pool of knowledge grows; the per-token compute does not.

That is the trick, and it is the whole reason the label and the invoice diverge.

How K3's router actually works, concretely

Picture a token, say, the word "photosynthesis", arriving at one of K3's mixture-of-experts layers. Sitting in that layer are 896 expert sub-networks. In front of them is the router: a small neural network whose only job is to score how relevant each expert is for this specific token.

The router produces 896 scores. K3 keeps the top 16, normalises them, and sends "photosynthesis" through only those 16 experts. Their outputs get mixed together, weighted by the router's scores, and that mixture becomes the layer's answer. The other 880 experts, for this token, do nothing.

The next token might route to a completely different 16. Over a long document, most of the experts get used — but never all at once, and never for one token.

The active-parameter count of 104 billion is what you get when you add up the router, the 16 experts that fired, and the parts of the network that are dense (attention layers, embeddings, and so on — the plumbing that every token has to pass through regardless).

What this changes in how you read the release

The compute story. When Moonshot claims roughly a 2.5× scaling efficiency improvement over Kimi K2 in its technical report, a figure reported by Roo's Newsletter and not independently replicated, the claim is not that they trained a smarter model per FLOP.² It is that they got more total-parameter headroom for a given per-token compute budget by pushing sparsity harder. K2 activated 8 of 384 experts (roughly 2%). K3 activates 16 of 896 (roughly 1.8%). More specialists, similar fraction firing, more total knowledge stored per token of compute spent.

The hardware story. The Hugging Face repository is 1.56 TB across 118 files, per Digital Applied's account of the HF API.¹ That is the download bill. The in-memory footprint at MXFP4 quantisation is roughly 1.4 TB — a distinct figure that early coverage sometimes conflated with the download size. Neither number fits on a single GPU. To actually run K3 locally you need multi-GPU inference infrastructure, which is why Together AI and Modal announced hosted access on release day, per explainX's coverage.³ "Open weights" here does not mean "runs on your laptop"; it means "you can, in principle, host it yourself if you have the racks."

The benchmark story. On the Artificial Analysis Intelligence Index, per Digital Applied, K3 sits fourth overall behind Claude Fable 5 and GPT-5.6 Sol, and first among open-weight models.¹ Moonshot's own technical report, as summarised by Roo's Newsletter, acknowledges that K3's general user experience still trails those two closed frontier models overall, even where K3 leads on individual benchmarks like SWE-Marathon.² The point is not that K3 is behind — it is that "2.8 trillion" is the wrong frame for expecting it to be ahead. Total parameters are storage capacity. They are not intelligence.

A subtle thing that trips people up

A larger context window is not automatically better performance. On the BrowseComp benchmark, per Roo's Newsletter's read of the technical report, K3 scores 91.2 when the input is compacted down to 300K tokens and 90.4 when the full 1,048,576-token window is used.² The bigger window scored lower on the same task.

This is worth sitting with. Context window size is a spec — how much you can, in theory, cram in. Whether the model actually uses it well is a separate question, and often the practical move is to compact aggressively before you feed it in. The 1M number sells the release; the 300K number is what the benchmark actually used to hit its top score.

The mechanism, in one line to take away

Kimi K3 stores knowledge like a library with 896 specialists on staff, and for every question that walks in, exactly 16 of them answer while the other 880 keep reading. The library is enormous. The consultation is not.

Glossary

Mixture-of-experts (MoE) A model architecture where many smaller specialist sub-networks sit behind a router that picks which few to use for each token.

Sparse activation The property of a network where only a small fraction of parameters do work for any given input.

Router The small network in an MoE layer that scores experts and picks the top few for each token.

Active parameters The count of parameters that actually participate in producing each token, as opposed to the total in the model.

Dense model A model where every parameter runs for every token, the opposite of sparse.

MXFP4 A 4-bit floating-point weight format; K3 was trained aware of this format rather than compressed into it after training.

Context window The maximum number of tokens the model can read at once as input.

Footnotes and links

Further reading

• Kimi Team, "Kimi K3: Open Frontier Intelligence," alphaXiv listing, 27 July 2026. https://www.alphaxiv.org

Footnotes

1. Digital Applied Team, "Kimi K3 Open Weights Shipped: What the Licence Says," Digital Applied, 27 July 2026. https://www.digitalapplied.com/blog/kimi-k3-open-weights-shipped-license-restrictions-2026 ↩ ↩² ↩³

2. Roo's Newsletter, "Kimi K3 Open Weights Are Live: 2.8T Parameters, 1M Context, and a License Nobody Actually Read," 27 July 2026. https://roo.beehiiv.com/p/kimi-k3-open-weights-license-benchmarks ↩ ↩² ↩³

3. explainX, "Kimi K3 Open Weights Are Live — 2.8T Parameters, Day-0 on Together and Modal," 27 July 2026. https://explainx.ai/blog/kimi-k3-open-weights-2-8-trillion-parameters-july-2026 ↩