Moonshot AI just released Kimi K3. It is a 2.8-trillion-parameter model with native vision and a 1-million-token context window. Moonshot calls it the world’s first open 3T-class model.

What is Kimi K3?

Kimi K3 is a sparse Mixture-of-Experts (MoE) model built on two architectural updates. Those are Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). Both change how information flows across sequence length and model depth. K3 targets long-horizon coding, knowledge work, and reasoning.

Moonshot team states K3 is the first open model to reach 2.8 trillion parameters. For nine of the past twelve months, Kimi models set the upper bound of open-model sizes.

Moonshot is also direct about where K3 sits. Overall performance still trails the most powerful proprietary models, Claude Fable 5 and GPT 5.6 Sol. Across Moonshot’s own evaluation suite, K3 consistently outperformed other tested models.

The Architecture Underneath

Kimi Delta Attention (KDA) is a hybrid linear attention mechanism. Moonshot states it enables up to 6.3x faster decoding in million-token contexts.

AttnRes works along the other axis, which is depth. It selectively retrieves representations across depth rather than accumulating them uniformly. Moonshot states AttnRes delivers roughly 25% higher training efficiency at under 2% additional cost.

Sparsity is the third lever. K3 uses Stable LatentMoE, effectively activating 16 of 896 experts. At that sparsity, routing and optimization become first-order challenges. Quantile Balancing derives expert allocation directly from router-score quantiles. That eliminates heuristic updates and a sensitive balancing hyperparameter. Per-Head Muon extends Muon by optimizing attention heads independently. Sigmoid Tanh Unit (SiTU) and Gated MLA improve activation control and attention selectivity respectively.

Refined training and data recipes accompany those structural changes. Together they yield roughly 2.5x better overall scaling efficiency than Kimi K2.

Those choices carry into serving. K3 applies quantization-aware training from the SFT stage onward. It uses MXFP4 weights with MXFP8 activations for broad hardware compatibility. Moonshot team recommends supernode configurations with 64 or more accelerators. Because KDA poses new challenges for prefix caching, Moonshot contributed an implementation to vLLM.