Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter) Your Dispatcher Target Device General Devices (CPU/GPU/FPGA) Mark supported Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter) Your Dispatcher Target Device General Devices (CPU/GPU/FPGA) Mark supported Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter) Your Dispatcher Target Device General Devices (CPU/GPU/FPGA) Mark supported

goingUnified Runtime For Heterogeneous Devices CUDA Driver NPU Driver Device Drivers External Runtimes NPUModule CUDAModule TFModule tvm::runtime::Module GetFunction(string) -> tvm::runtime::PackedFunc nc SaveToBinary/LoadFromBinary Runtime Module Interface SubclassesUnified Runtime Benefit mod.export_library("mylib.so") Unified library packaging Free API (Py/Java/Go) lib = tvm.module.load("mylib remote_b)Virtual Machine: Supporting Dynamic Workload Dynamic shape workloads More runtime objects: Arrays, Tuples, Trees, ADTs Minimum runtime for dynamic models Credit: Jared Roesch, Haichen Shen et.aluTVM: TVM

truncating division. e Unified Object and Node system for TVM runtime o Lays groundwork forimproved multi-language support for expPosing runtime, and |IRs. QQ octoML Unified Object Protocol vm::Object NDArray | Rd | tuplelclosure AST Nodes Cross language suppPort Easy to introduce new runtime objects (trees, graphs) Direct access from other languages QQ octoML HTVM Overview *。 Plug directly emit code for microcontrollers that is device- agnostic AuroTYM QQ octoML AutoTVM on HTVM DTYM Runtime send program 较 ,we 人 Interace Optimize TVM operators on microcontrollers by making use of

neural networks >> 4© Copyright 2018 Xilinx Inference Flow >> 5 MxNet CPU Layers FPGA Layers Runtime Image Model Weights Calibration Set Quantizer Compiler Tensor Graph Optimization Framework Tensor 11 Calls XDNN’s TVM registered function to access the FPGA runtime APIs© Copyright 2018 Xilinx Registering TVM op in Python at runtime File contrib_xlnx.py: … @tvm.register_func("tvm.accel.accel_fused") contains multiple stages, performance limited by slowest one ˃ Performance results based on Xilinx own runtime pipeline available in github (https://github.com/Xilinx/ml-suite/blob/master/examples/deployment_modes/mp_classify

dependent: arange, nms, etc. ○ Control flow: concatenate within a while loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates. All TVM ● Support Any-dim in typing ● Use shape function to compute the type at runtime ● Virtual machine as a new runtime for Relay ● Dynamic codegen (WIP) ○ Kernel dispatch for a single op ○ Graph Fit for operator such as conv2d_NCHWc. Graph tuning is well defined for each subgraph. 3. Avoid runtime layout tracking system for operator requires layout transformation to optimize.© 2019, Amazon Web

站 Compile | libtvm_hexagon_runtime.so Alios TVM @ Hexagon DSP 。 Compute Kernel Offload to DSP ，loop nests marked as pipeline 。， Implement complete Hexagon runtime based on community PR. ADSPRPC inherits LLVM and could generate HVX instruction 。， Add one Hexagon runtimes named as libtvm_hexagon_runtime.so to support parallel. 。 Could run end-to-end TFLite Mobilenet V2 quantized model on Simulator

FC layers - 24kHz sampling frequency requires 40us sampling net runtime - First PyTorch model used a 3,400us sampling net runtime Image from LPCNetExit, Pursued By A Bear - 3400us (baseline), 40us

ACL/CMSIS-NN kernels into TVM? ○ Implement Arm NN generic backend in TVM for more flexibility with the runtime plugins? ○ Integrate TVM codegen into Arm NN? ● CI and benchmark testing for TVM on member hardware

Graph Optimizations -Param TvM Tensor Operators & Runtime Property Registr \L Compiler Toolchain 于 TVM TOPI Schedule Primitives & Optimizations Symbols