rights reserved. Presenter: Haichen Shen, Yao Wang Amazon SageMaker Neo, Deep Engine Science Dynamic Model in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Models with models© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support dynamic model in TVM ● Support Any-dim in typing ● Use shape function to compute the type at runtime ● Virtual input_name = "data" input_shape = [tvm.relay.Any(), 3, 224, 224] dtype = "float32" block = get_model('resnet50_v1', pretrained=True) mod, params = relay.frontend.from_mxnet(block, shape={input_name:

Change Happening Faster Than Ever? Yes, It Is • AI User + Usage + CapEx Growth = Unprecedented • AI Model Compute Costs High / Rising + Inference Costs Per Token Falling = Performance Converging + Developer 2/24 2/25 4/25 75% 60% 10% 21% 15% 0% Details on Page 293 USA – LLM #1 China USA – LLM #2 AI Model Compute Costs High / Rising + Inference Costs Per Token Falling = Performance Converging + Developer Richard Hirsh; John McCallum; OpenAI Details on Page 138 0 Years 72 Years Electric Power Computer Memory AI Inference AI Monetization Threats = Rising Competition + Open-Source Momentum + China’s Rise

FABRIC IMG RD SCHEDULER WEIGHTS RD SCHEDULER PE Array PE PE PE PE DISPATCHER ... EXTERNAL MEMORY INSTR FETCHER DECODER REG MAP WB WR SCHEDULER CTRL SIGNALS MISC CALC AVG POOL MAX POOL >> 4© Copyright 2018 Xilinx Inference Flow >> 5 MxNet CPU Layers FPGA Layers Runtime Image Model Weights Calibration Set Quantizer Compiler Tensor Graph Optimization Framework Tensor Graph to ins, outs: tvm.call_packed('tvm.accel.accel_fused', attrs['path'], attrs['output_layout'], attrs['model_name'], outs[0], *ins ), name=name) return out >> 10© Copyright 2018 Xilinx Example of FPGA node

Calling Julia Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 30.5 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 30.6 Working with Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 32.5 Memory allocation analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 32.6 External . . . . . . . . . . . . . . . . . 416 34.3 Measure performance with @time and pay attention to memory allocation . . . . . . 417 34.4 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Calling Julia Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 30.5 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 30.6 Working with Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 32.5 Memory allocation analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 32.6 External . . . . . . . . . . . . . . . . . 416 34.3 Measure performance with @time and pay attention to memory allocation . . . . . . 417 34.4 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Calling Julia Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 31.5 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 31.6 Working with Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432 33.5 Memory allocation analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 33.6 External . . . . . . . . . . . . . . . . . 444 35.3 Measure performance with @time and pay attention to memory allocation . . . . . . 445 35.4 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Calling Julia Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 31.5 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 31.6 Working with Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432 33.5 Memory allocation analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 33.6 External . . . . . . . . . . . . . . . . . 444 35.3 Measure performance with @time and pay attention to memory allocation . . . . . . 445 35.4 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Calling Julia Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 31.5 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 31.6 Working with Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432 33.5 Memory allocation analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 33.6 External . . . . . . . . . . . . . . . . . 444 35.3 Measure performance with @time and pay attention to memory allocation . . . . . . 445 35.4 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Calling Julia Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408 31.5 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 31.6 Working with Wall-time Profiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437 33.6 Memory allocation analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 33.7 External Type inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459 35.4 Memory management and arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 35.5 Execution latency

Calling Julia Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408 31.5 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 31.6 Working with Wall-time Profiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437 33.6 Memory allocation analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 33.7 External Type inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459 35.4 Memory management and arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 35.5 Execution latency