Momentum Performance, 16-bit FLOP/s +150% / Year Enabled by 1.6x annual growth in chips per cluster and 1.6x annual growth in performance per chip Performance of Leading AI Supercomputers (FLOP/s) ecosystem. Many open-source tools still lack the brand power, plug-and-play user experience (UX), and managed services that drive adoption among consumers and large organizations. But as the cost-performance delivers advanced AI chip ‘cluster’ to Chinese clients cut off from Nvidia’ (4/29/25) Huawei has started the delivery of its advanced artificial intelligence chip ‘cluster’ to Chinese clients who are

of FlashAttention-2 (Dao, 2023). We conduct all experiments on a cluster equipped with NVIDIA H800 GPUs. Each node in the H800 cluster contains 8 GPUs connected using NVLink and NVSwitch within nodes. attain a relatively high Model FLOPs Utilization (MFU). During our practical training on the H800 cluster, for training on each trillion tokens, DeepSeek 67B requires 300.6K GPU hours, while DeepSeek-V2

TPU-like Specialized Accelerators Tensor Compute Primitives Unified Buffer Acc FIFO Explicitly Managed Memory Subsystem TPUsTensorization Challenge Compute primitives scalar vector tensor Challenge: