Richard Hirsh; John McCallum; OpenAI Details on Page 138 0 Years 72 Years Electric Power Computer Memory AI Inference AI Monetization Threats = Rising Competition + Open-Source Momentum + China’s Rise its 365 product suite 3/23: Anthropic releases Claude, its AI assistant focused on safety & inter- pretability 3/24: USA Department of Homeland Security unveils its AI Roadmap 11/23: 28 countries, including USA, EU members & China, sign Bletchley Declaration on AI Safety 4/24: Meta Platforms releases its open- source** Llama 3 model with 70B parameters

conversational sessions, which encompass various domains such as math, code, writing, reasoning, safety, and more, to perform Supervised Fine-Tuning (SFT) for DeepSeek-V2 Chat (SFT). Finally, we follow the KV joint compression in MLA reduces the KV cache. Moreover, in order to reduce the activation memory during training, we also perform 7 low-rank compression for the queries, even if it cannot reduce relatively few activated parameters, and a portion of the operators are recomputed to save acti- vation memory, it can be trained without the necessity of tensor parallelism, thereby decreasing the communication

improvements. That means shipping updates regularly, getting feedback, and improving performance and safety at every step. The result: users access new advancements in AI early and often—and your feedback AI in the EnterpriseLesson 1 Start with evals How Morgan Stanley iterated to ensure quality and safety As a global leader in financial services, Morgan Stanley is a relationship business. Not surprisingly theme: AI deployment benefits from an open, experimental mindset, backed by rigorous evaluations, and safety guardrails. The companies seeing success aren’t rushing to inject AI models into every workflow

tall is the Empire State Building?” is an 
 off-topic user input and would be flagged as irrelevant. Safety classifier Detects unsafe inputs (jailbreaks or prompt injections) 
 that attempt to exploit system vulnerabilities. We’ve found the following heuristic to be effective: 01 Focus on data privacy and content safety 02 Add new guardrails based on real-world edge cases and failures you encounter 03 Optimize for both such as jailbreak prevention, relevance validation, keyword filtering, blocklist enforcement, or safety classification. For example, the agent above processes a math question input optimistically until

model to create a structure and limit hallucinations. System prompts can also be really useful for safety and toxicity. To control the output, simply add an additional line to your prompt like: ‘You should instructions, clearly stating what you want the model to do and only use constraints when necessary for safety, clarity or specific requirements. Experiment and iterate to test different combinations of instructions

Tuning），如下图11所示。通用强化学习训练过 程后，使得 R1 不仅在推理任务中表现卓越，同时在非推理任务中也表现出 色。但由于其能力拓展至非推理类应用，因此在这些应用中引入了帮助性 （helpfulness）和安全性（safety）奖励模型（类似于 Llama 模型），以优化 与这些应用相关的提示处理能力。 DeepSeek-R1 是训练流程的终点，结合了 R1-Zero 的推理能力和通用强化 学习的任务适应能力，成为一个兼具强推理和通用能力的高效

part of the systeml e Haichen and | will discuss more details at TVMConf. Oo oo QQ octoML 11 VM Memory Planning e Recently shipped a first version fn enain(0) -> Tensor[tk,)，f32] { ofdynamicmemory Planmng Let t2 3 memory planning,， storage Let s = alLLoc_storage(40，64，f32) ; Tet outl = attoc_tensor(s，(19,)，f32); coalescing, memory re-use for invoke_ l，t2)，(outl,))3 Out1l loops, and offloading dynamic } allocation to devices. QQ octoML VM Memory Abstractions Old New t1: Tensor t1: Tensor

VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 5 Software - CMA Contiguous Memory Allocation – Linux Kernel DEPLOY VTA ON INTEL FPGA https://pynq.readthedocs.io/en/v2.0/pynq_package/pynq 08.02_pr.tar.gz©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 6 Software - CMA Contiguous Memory Allocation – Linux Kernel Module DEPLOY VTA ON INTEL FPGA Setup Environment Variables Navigate INTERNATIONAL INDUSTRIES, INCORPORATED 7 Software - Driver Cyclone V & Arria V SoC HPS Physical Memory Map DEPLOY VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 8 Hardware Configure

TensorCore Intrinsics 。Authored by @Hzfengsy 。 Intrinsics: tvm_load_matrix_sync tvm_mma_sync … “New Memory Scopes: wmma.matrix_a/b, accumulator 。Tensorization on warp level schedule Motivation load/store for higher bandwidth utilization 。Double buffer to hide memory load latency 。 storage align to reduce bank conflicts of shared memory 。 Virtual threads for data reuse (on going) Performance on V100

FABRIC IMG RD SCHEDULER WEIGHTS RD SCHEDULER PE Array PE PE PE PE DISPATCHER ... EXTERNAL MEMORY INSTR FETCHER DECODER REG MAP WB WR SCHEDULER CTRL SIGNALS MISC CALC AVG POOL MAX POOL aster/examples/deployment_modes/mp_classify.py) Streamlined multi-process pipeline using shared memory Usually need >4 Pre-Process cores running to keep up with FPGA ˃ TVM pipeline needed. CPU/FPGA