Machine Learning Lecture 10: Neural Networks and Deep Learning Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2018 Deep Feedforward usually a highly non-linear function • Feedforward networks are of extreme importance to machine learning practioners • The conventional neural networks (CNN) used for object recognition from photos are units), and output layer 7 / 19 Neural Feedforward Networks (Contd.) • We approximate f ∗(x) by learning f(x) from the given training data • In the output layer, f(x) ≈ y for each training data, but the

Chapter 3 - Learning Techniques “The more that you read, the more things you will know. The more that you learn, the more places you'll go.” ― Dr. Seuss Model quality is an important benchmark to evaluate evaluate the performance of a deep learning model. A language translation application that uses a low quality model would struggle with consumer adoption because it wouldn’t serve its intended purpose flexibility to trade off some quality for smaller footprints. In the first chapter, we briefly introduced learning techniques such as regularization, dropout, data augmentation, and distillation to improve quality

Machine Learning Pytorch Tutorial TA : 曾元（Yuan Tseng） 2022.02.18 Outline ● Background: Prerequisites & What is Pytorch? ● Training & Testing Neural Networks in Pytorch ● Dataset & Dataloader ● Tensors link3 2. Deep Learning Basics ■ Prof. Lee’s 1st & 2nd lecture videos from last year ■ ref: link1, link2 Some knowledge of NumPy will also be useful! What is PyTorch? ● An machine learning framework in computing with more cores for arithmetic calculations ○ See What is a GPU and do you need one in deep learning? Tensors – Gradient Calculation >>> x = torch.tensor([[1., 0.], [-1., 1.]], requires_grad=True)

Advanced Learning Techniques “Tell me and I forget, teach me and I may remember, involve me and I learn.” – Benjamin Franklin This chapter is a continuation of Chapter 3, where we introduced learning techniques techniques. To recap, learning techniques can help us meet our model quality goals. Techniques like distillation and data augmentation improve the model quality, without increasing the footprint of the model this chapter by presenting self-supervised learning which has been instrumental in the success of natural language models like BERT. Self-Supervised learning helps models to quickly achieve impressive

variety of techniques in the last few chapters to improve efficiency and boost the quality of deep learning models. These techniques are just a small subset of the available techniques. It is often tedious these four options to make an informed decision. Blessed with a large research community, the deep learning field is growing at a rapid pace. Over the past few years, we have seen newer architectures, techniques performance benchmarks higher. Figure 7-1 shows some of the choices we face when working on a deep learning problem in the vision domain for instance. Some of these choices are boolean, others have discrete

Introduction to Efficient Deep Learning Welcome to the book! This chapter is a preview of what to expect in the book. We start off by providing an overview of the state of deep learning, its applications, and motivation behind seeking efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient models & layers even if you just read this chapter, you would be able to appreciate why we need efficiency in deep learning models today, how to think about it in terms of metrics that you care about, and finally the tools

or quality, we should consider employing suitable efficient architectures. The progress of deep learning is characterized by the phases of architectural breakthroughs to improve on previous results and (CNNs) were another important breakthrough that enabled learning spatial features in the input. Recurrent Neural Nets (RNNs) facilitated learning from the sequences and temporal data. These breakthroughs optimization experience using these efficient layers and architectures. Let’s start our journey with learning about embeddings in the next section. Embeddings for Smaller and Faster Models We humans can intuitively

make it shorter.” Blaise Pascal In the last chapter, we discussed a few ideas to improve the deep learning efficiency. Now, we will elaborate on one of those ideas, the compression techniques. Compression gentle introduction to the idea of compression. Details of quantization and its applications in deep learning follow right after. The quantization section delves into the implementation details using code samples might lead to degradation in quality. In our case, we are concerned about compressing the deep learning models. What do we really mean by compressing though? As mentioned in chapter 1, we can break down

them, with an eye towards conceptual understanding as well as practically using them in your deep learning models. We start with sparsity. If your goal was to optimize your brain for storage, you can often different possible methods of picking the connections and nodes to prune, and how to prune a given deep learning model to achieve storage and latency gains with a minimal performance tradeoff. Next, the chapter effectiveness. Later on in this chapter, we have a project that relies on it for sparsifying a deep learning model. The authors of the Optimal Brain Damage (OBD) paper approximate the saliency score using

Lecture 1: Overview 1 About the Course 2 Machine Learning: What and Why? 3 Categories of Machine Learning 4 Some Basic Concepts of Machine Learning Feng Li (SDU) Overview September 6, 2023 2 / 57 Course Information We will investigate fundamental concepts, techniques and algorithms in machine learning. The topics include linear regression, logistic re- gression, regularization, Gaussian discriminant Readings Hang Li, Statistical Machine Learning (2nd Ed.), The Tsinghua Press, 2019 Zhihua Zhou, Machine Learning, Tsinghua Press, 2016 Tom M. Mitchell, Machine Learning (1st Ed.), China Machine Press, 2008