Leveraging the power of C++ for efficient machine learning on embedded devices Adrian Stanciu adrian.stanciu.pub@gmail.com CppCon, 2023 1 / 50About me ◮ I am a software engineer from Romania ◮ I have Image classification ◮ Hand gesture recognition ◮ Summary ◮ Q&A 4 / 50Motivation 5 / 50Machine Learning (ML) ◮ Subfield of Artificial Inteligence (AI) ◮ Enables computers to learn from data and then consumption ◮ May have real-time performance constraints 7 / 50Machine learning on embedded devices ◮ Alternative to cloud-based machine learning ◮ Advantages: ◮ Real-time processing ◮ Low latency ◮ Reduced bandwidth

micrograd++: A 500 line C++ Machine Learning Library Gautam Sharma Independent Researcher gautamsharma2813@gmail.com Abstract—micrograd++ is a pure C++ machine learning li- brary inspired by Andrej Karpathy’s for building and training machine learning models. By leveraging the performance efficiency of C++, micro- grad++ offers a robust solution for integrating machine learning capabilities directly into C++-based Traditionally, all machine learning libraries are extremely bulky and very hard to integrate as third party dependencies. This aspect scares practitioners to adopt a C++ based machine learning library for prototyping

Intelligence,’ a term he coined 1/62: Arthur Samuel, an IBM computer scientist, creates a self-learning program that proves capable of defeating a top USA checkers champion AI ‘Winter1’ (1967-1996) Trending = Unprecedented37 Machine-Learning Model* Trending = In 2015... Industry Surpassed Academia as Data + Compute + Financial Needs Rose *Machine Learning = A subset of AI where machines learn AI Index data provider, uses the term ‘notable machine learning models’ to designate particularly influential models within the AI/machine learning ecosystem. Epoch maintains a database of 900 AI models

Part 5: The Third Way – The Technical Practices of Continual Learning and Experimentation; 1. Introduction a. Goal – practices to enable learning as quickly, frequently, cheaply, and as soon as possible safety, continuous improvement, and learning ii. Create mechanism to rapidly spread learning throughout the organization 2. Ch. 19 – Enable and Inject Learning into Daily Work a. Complex systems are Monkey; they architected for failure, tested for failure, and evolved beyond it b. ESTABLISH A JUST, LEARNING CULTURE i. Unjust responses to incidents 1. Impede safety 2. Promote fear over mindfulness

Building correct, secure, and safe systems Planning for the future Agenda Resources for learning modern C++ 0501 Obtaining tools for your platform 02 03 04 Code reuse through libraries Building correct, secure, and safe systems Planning for the future Agenda Resources for learning modern C++ 05Tools for your platform Build & project systems Editors & IDEs Compiler toolchainsTools Building correct, secure, and safe systems Planning for the future Agenda Resources for learning modern C++ 0501 Obtaining tools for your platform 02 03 04 Code reuse through libraries

TVM: Where are we going Tianqi ChenCurrent Deep Learning Landscape Frameworks and Inference engines DL Compilers Kenrel Libraries Hardware CuDNN NNPack MKL-DNN Hand optimized Open source, automated automated end-to- end optimization framework for deep learning.TVM Stack High-Level Differentiable IR Tensor Expression and Optimization Search Space LLVM, CUDA, Metal VTA Edge FPGA Cloud FPGA FPGA ASIC Optimization AutoTVM Device FleetExisting Deep Learning Frameworks High-level data flow graph Hardware Primitive Tensor operators such as Conv2D eg. cuDNN Offload to heavily optimized

researching on computers and programming and how to practically apply it well for several years, still learning and enjoying the journey. • I am passionate about energy efficient computing and making the most the use of C level constructs, but when wrapped in C++ containers, types, with the use of RAII Learning/Teaching C++ • Now that we have talked a little about the “Why” I would like to touch on the “How” This learning is gradual and happens step by step. • Adults Help to Improve the Learning of Children by Stepping Down A level Closer to Children Through “Baby Talk” How Does this Map to Learning C++

the various powers that are given to us from the ground up. Consider this talk your one stop for learning all of the great things about functions! We'll start with a basic function example, identifying (1/3) 17 ● I remember learning about loops ○ (Using goto and do/loop) ■ Wow -- I can save myself repeating typing of code!A Monumental Moment! (2/3) 18 ● I remember learning about loops ○ (Using something different than the math functions I’m learning in school ■ Sounds like I can do some interesting stuff!A Monumental Moment! (3/3) 19 ● I remember learning about loops ○ (Using goto and do/loop)

practices. Extract real-time insights about software development life- cycle.68 Aurora Innovation Learning what to test Good tests verify a system expectation that is prone to change. Good tests are the Scope Con�dence By Etienne Jong69 Aurora Innovation Learning how to test Expressive Good tests are easy to read and effective way of learning business logic. Apply the same code hygiene to your tests production code. Measure test coverage in terms of veri�ed business requirements.70 Aurora Innovation Learning when to run each test Reliability Fast Feedback Optimize for return on investment. Avoid reusing

Pointers are scary. Unfortunately that previous statement is what many beginners take away when first learning about pointers and the C++ language. In this talk, we will discuss the low level foundations of find pointers are not scary, but just another tool we can use in our programmers’ toolbox. After learning the foundations, we are then going to discuss some of the pitfalls of pointers (e.g. nullptr’s ● When learning pointers, it’s often useful to draw (on pen and paper) our memory ○ Let’s represent our variables as boxes for now.Visualizing our first program (2/5) 23 ● When learning pointers