All rights reserved. Amazon Trademark Deploy GluonCV Models GluonCV Models MXNet Computational Graph Json Acyclic Graph Export As-is Optimize with TVM© 2019, Amazon Web Services, Inc. or its Affiliates its Affiliates. All rights reserved. Amazon Trademark Like GluonCV? Go build! https://gluon-cv.mxnet.io https://github.com/dmlc/gluon-cv© 2019, Amazon Web Services, Inc. or its Affiliates. All rights

ingests a FP32 graph and a small dataset • Finds suitable quantization scale • Produces a quantized graph • Compiling Pre-quantized models – QNN Dialect • TVM ingests a pre-quantized graph in TFLite or or MxNet • Use high-level wrapper ops of QNN dialect© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TVM Overview Framework Graph Mxnet TF …. parsers Relay Graph Target-independent Target-independent Relay passes Target-optimized graph Target-dependent Relay passes Intel x86 ARM CPU Nvidia GPU ARM GPU Schedule templates written in TVM Tensor IR .. More targets AutoTVM – Tuning

can run any models Your compiler (TVM) supports multiple frontends (e.g., TensorFlow, PyTorch, MXNet) Non Maximum Suppression ResNet-50 Dense Your Chip Your Chip© 2019, Amazon Web Services, Inc System Overview Relay IR Graph Annotation with Your Annotator Graph Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter) Mark supported operators or subgraphs 1. Implement an operator-level annotator, OR 2. Implement a graph-level annotator© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Option 1:

Inference Flow >> 5 MxNet CPU Layers FPGA Layers Runtime Image Model Weights Calibration Set Quantizer Compiler Tensor Graph Optimization Framework Tensor Graph to Xilinx Tensor Graph Frontend Deep https://github.com/xilinx© Copyright 2018 Xilinx TVM as Unified ML Front End >> 6 Relay (and NNVM) Graph Parser XIR Compiler Quantizer Partitioner @relay.transform.module_pass(opt_level=4) class AccelModule:© supported/not supported, pattern matching graph colorization - Choices how to partition especially for multi-branch networks (i.e. YOLOv3, SSD)© Copyright 2018 Xilinx TVM Graph Partitioning/Fusion >> 8 Subgraph

dependent: arange, nms, etc. ○ Control flow: concatenate within a while loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates new runtime for Relay ● Dynamic codegen (WIP) ○ Kernel dispatch for a single op ○ Graph dispatch for a (sub-)graph In collaboration with Jared Roesch, Zhi Chen, Wei Chen© 2019, Amazon Web Services, implement© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dispatch a Whole Graph Resnet Data -> (Any, 3, 224, 224) Dispatch Tree Resnet_copy0 Resnet_copy1 ... 1 <= bs < 17 17

Hexagon DSP 人NiOS ! 驱动万物知 Tensorflow deploy.so / deploy.json / deploy.bin | NNVM / Relay 让 Graph Optimization 站 站 Compile | libtvm_hexagon_runtime.so Alios TVM @ Hexagon DSP 。 Compute Kernel 6 4 2.353 2. ， 曾硬证 0 Mobilenet 1.0 densenet121 量TVM (with Auto Tuning) 目MXNet+ TensorRT 目TVM +TensorRT AiiOS ! 驱动万物智能 THANKS9 Ali0S ! 驱动万物智能

中国人工智能框架市场调研报告，艾瑞咨询研究院自主研究及绘制。 AI框架的开源趋势 AI现已经成为最受关注的开源领域，部分得益于海外厂商对AI框架开源的促进。2015年是AI开源的重要一年，TensorFlow、 MXNet、Keras于该年先后开源，PyTorch紧随其后，在2016年也宣布开放。这一系列动作使得国内AI开发者可无障碍获取成熟的 海外框架，海外框架由此获得先发优势。目前来看，海外框架在国内的使用率仍 大数据 编程语言 AI 2015 PyTorch 34% TensorFlow 30% MindSpore 11% PaddlePaddle 11% OneFlow 3% MXNet 2% MegEngine 2% Jittor 1% 其他 6% 2016 2020 AI产业发展释放底层开发需求，国产开源框架不惧挑战奋起直追 中国开发者主流人工智能框架使用率排名

GPU 加速，提供极速特征向量匹配以及多维度数据联合查询（特 征、标签、图片、视频、文本和语音等联合查询）功能，并且支持自动分库分表和多副本，能对 接 TensorFlow、PyTorch 和 MxNet 等 AI 模型，可实现百亿特征向量的秒级查询。Milvus 于 2019 年 10 月在 GitHub 上开源，Stars 和 Docker Pulls 数量持续高速增长，2021 年 6 splitting and multi-copy support and can connect with AI models such as TensorFlow, PyTorch, and MxNet to achieve ten billion feature vectors in seconds. Milvus opened source on GitHub in October 2019

GPU 加速，提供极速特征向量匹配以及多维度数据联合查询（特征、标签、图片、视 频、文本和语音等联合查询）功能，并且支持自动分库分表和多副本，能对接 TensorFlow、PyTorch 和 MxNet 等 AI 模型，可实现百亿特征向量的秒级查询。Milvus 于 2019 年 10 月在 GitHub 上开 源，Stars 和 Docker Pulls 数量持续高速增长，2021 年 6 月达到

Integrated stereo/multi-view pipeline, Smooth Corrective modifier and new developmental dependency graph. 2.76 – November 2015: Pixar OpenSubdiv support, Viewport and File Browser performance boost, node to organize objects. Other improvements: Cycles, Modeling, Animation, Import/Export, Dependency Graph. 2.81 – November 2019: Revamped sculpting tools, Cycles OptiX accelerated rendering, denoising, dialog. Common Editor Keys These keys are shared across editors such as the 3D Viewport, UV and Graph editor. A Select all. Alt-A Select none. Ctrl- I Invert selection. H Hide selection. Alt-H Reveal