DeepSeek-V3 Technical Report

arXiv:2412.19437

DeepSeek-AI

DeepSeek-V3 is a strong Mixture-of-Experts language model with 671B total parameters and 37B activated per token. We describe Multi-head Latent Attention, an auxiliary-loss-free load balancing strategy, and multi-token prediction training objectives that improve efficiency and quality.