INDUSTRIES, INCORPORATED ACCELERATED VISUAL PERCEPTION LIANGFU CHEN 11/16/2019 DEPLOY VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 2 Moore’s Law is Slowing Down MOTIVATION©2019 Terasic DE10-Nano DEPLOY VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 5 Software - CMA Contiguous Memory Allocation – Linux Kernel DEPLOY VTA ON INTEL FPGA https://pynq.readthedocs INCORPORATED 6 Software - CMA Contiguous Memory Allocation – Linux Kernel Module DEPLOY VTA ON INTEL FPGA Setup Environment Variables Navigate to 3rdparty/cma and build kernel module Copy kernel module

Target-independent Relay passes Target-optimized graph Target-dependent Relay passes Intel x86 ARM CPU Nvidia GPU ARM GPU Schedule templates written in TVM Tensor IR .. More targets AutoTVM – Tuning Target-independent Relay passes Target-optimized Int8 Relay Graph Intel x86 schedule ARM CPU schedule Nvidia GPU schedule ARM GPU schedule Relay Int8 Graph Target-dependent Relay layout opt© 2019 its Affiliates. All rights reserved. Outline • QNN Dialect • Design • Operators • Results on Intel Cascade Lake© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Quantized Operators

人 人 e 人 e@ TVM Q@ AliOs Overview TVM @ AliOs ARM CPU TVM @ AliOos Hexagon DSP TVM @ Alios Intel GPU Misc /NiiOS ! 驱动万物智能 PART ONE TVM Q@ AliOs Overview AiOS 1驱动万物智能 AliOs overview 。 AliOs (www AN 2X MobilenetV2 TFLite 1.34X MobilenetV2 QNNPACK AliOs @ Roewe RX5 MAX OpenVINO @ Intel GPU AliDS AR-Nav Product @ SUV Release and adopt TVM (Apollo Lake Gold) Model 1.6X Intel AliOs TVM Arch Model 。 Facelandmark Pedestrian & Vehicle Detection Voice-GUI Gesture Lanenet NLU DMS FacelD Multimodal Interection CPU (ARM、Intel) 1驱动万物智能 Accelerated

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon/Intel Confidentia Presenter: Zhi Chen, Cody Yu Amazon SageMaker Neo, Deep Engine Science Bring Your Own Codegen to TVM Chip© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Example showcase: Intel MKL-DNN (DNNL) library 1. Import packages import numpy as np from tvm import relay 2. Load a pretrained Relay Runtime (VM, Graph Runtime, Interpreter) Your Dispatcher Target Device General Devices (CPU/GPU/FPGA) Mark supported operators or subgraphs 1. Implement an operator-level annotator, OR 2. Implement

Impressive61 NVIDIA AI Ecosystem Tells Over Four Years = >100% Growth in Developers / Startups / Apps Note: GPU = Graphics Processing Unit. Source: NVIDIA (2021 & 2025) NVIDIA Computing Ecosystem – 2021-2025, per Cloud vs. AI Patterns105 Tech CapEx Spend Partial Instigator = Material Improvements in GPU PerformanceNVIDIA GPU Performance = +225x Over Eight Years 106 1 GPT-MoE Inference Workload = A type of workload Source: NVIDIA (5/25) Performance of NVIDIA GPU Series Over Time – 2016-2024, per NVIDIA Tech CapEx Spend Partial Instigator = Material Improvements in GPU Performance Pascal Volta Ampere Hopper Blackwell

performance gain by autotuning 3. TVM can support many kinds of hardware platform: Intel/arm CPU, Nividia/arm GPU, VTA…5 �������������� 1. Get a .log file from the autotvm on Ubuntu 2. Use the .log

MPI.jl and Elemental.jl provide access to the existing MPI ecosystem of libraries. 4. GPU computing: The Julia GPU compiler provides the ability to run Julia code natively on GPUs. There is a rich ecosys- array operations distributed across workers, as outlined above. A mention must be made of Julia's GPU programming ecosystem, which includes: 1. CUDA.jl wraps the various CUDA libraries and supports compiling (GNU), others optionally permit placing hidden arguments directly after the character argument (Intel, PGI). For example, Fortran subroutines of the form subroutine test(str1, str2) character(len=*)

MPI.jl and Elemental.jl provide access to the existing MPI ecosystem of libraries. 4. GPU computing: The Julia GPU compiler provides the ability to run Julia code natively on GPUs. There is a rich ecosys- array operations distributed across workers, as outlined above. A mention must be made of Julia's GPU programming ecosystem, which includes: 1. CUDA.jl wraps the various CUDA libraries and supports compiling (GNU), others optionally permit placing hidden arguments directly after the character argument (Intel, PGI). For example, Fortran subroutines of the form subroutine test(str1, str2) character(len=*)

MPI.jl and Elemental.jl provide access to the existing MPI ecosystem of libraries. 4. GPU computing: The Julia GPU compiler provides the ability to run Julia code natively on GPUs. There is a rich ecosys- array operations distributed across workers, as outlined above. A mention must be made of Julia's GPU programming ecosystem, which includes: 1. CUDA.jl wraps the various CUDA libraries and supports compiling (GNU), others optionally permit placing hidden arguments directly after the character argument (Intel, PGI). For example, Fortran subroutines of the form subroutine test(str1, str2) character(len=*)

MPI.jl and Elemental.jl provide access to the existing MPI ecosystem of libraries. 4. GPU computing: The Julia GPU compiler provides the ability to run Julia code natively on GPUs. There is a rich ecosys- array operations distributed across workers, as outlined above. A mention must be made of Julia's GPU programming ecosystem, which includes: 1. CUDA.jl wraps the various CUDA libraries and supports compiling (GNU), others optionally permit placing hidden arguments directly after the character argument (Intel, PGI). For example, Fortran subroutines of the form subroutine test(str1, str2) character(len=*)