of FlashAttention-2 (Dao, 2023). We conduct all experiments on a cluster equipped with NVIDIA H800 GPUs. Each node in the H800 cluster contains 8 GPUs connected using NVLink and NVSwitch within nodes. attain a relatively high Model FLOPs Utilization (MFU). During our practical training on the H800 cluster, for training on each trillion tokens, DeepSeek 67B requires 300.6K GPU hours, while DeepSeek-V2

Momentum Performance, 16-bit FLOP/s +150% / Year Enabled by 1.6x annual growth in chips per cluster and 1.6x annual growth in performance per chip Performance of Leading AI Supercomputers (FLOP/s) delivers advanced AI chip ‘cluster’ to Chinese clients cut off from Nvidia’ (4/29/25) Huawei has started the delivery of its advanced artificial intelligence chip ‘cluster’ to Chinese clients who are