Chapter 3 - Learning Techniques “The more that you read, the more things you will know. The more that you learn, the more places you'll go.” ― Dr. Seuss Model quality is an important benchmark to evaluate evaluate the performance of a deep learning model. A language translation application that uses a low quality model would struggle with consumer adoption because it wouldn’t serve its intended purpose flexibility to trade off some quality for smaller footprints. In the first chapter, we briefly introduced learning techniques such as regularization, dropout, data augmentation, and distillation to improve quality

Advanced Learning Techniques “Tell me and I forget, teach me and I may remember, involve me and I learn.” – Benjamin Franklin This chapter is a continuation of Chapter 3, where we introduced learning techniques techniques. To recap, learning techniques can help us meet our model quality goals. Techniques like distillation and data augmentation improve the model quality, without increasing the footprint of the model this chapter by presenting self-supervised learning which has been instrumental in the success of natural language models like BERT. Self-Supervised learning helps models to quickly achieve impressive

variety of techniques in the last few chapters to improve efficiency and boost the quality of deep learning models. These techniques are just a small subset of the available techniques. It is often tedious these four options to make an informed decision. Blessed with a large research community, the deep learning field is growing at a rapid pace. Over the past few years, we have seen newer architectures, techniques performance benchmarks higher. Figure 7-1 shows some of the choices we face when working on a deep learning problem in the vision domain for instance. Some of these choices are boolean, others have discrete

Introduction to Efficient Deep Learning Welcome to the book! This chapter is a preview of what to expect in the book. We start off by providing an overview of the state of deep learning, its applications, and our motivation behind seeking efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient models & even if you just read this chapter, you would be able to appreciate why we need efficiency in deep learning models today, how to think about it in terms of metrics that you care about, and finally the tools

2016的目标：Web爬虫＋深度学习＋自然语言处理 ＝ ？ Microso� Apple AWS 今年最激动人心的事件？ 2016.1.28 “Mastering the game of Go with deep neural networks and tree search” 今年最激动人心的事件？ 2016年3月Alphago 4:1 击败李世石九段 人工智能 VS. 机器学习 VS. 深度学习 文的自动分类 半监督学习 - 介于监督学习和无监督学习之间，算法: Graph Inference 或者Laplacian SVM 强化学习- 通过观察来学习做成如何的动作, 算法：Q-Learning以及时间差学习 机器学习－ 方法及流程 输入特征选择 – 基于什么进行预测 目标 – 预测什么 预测功能 – 回归、聚类、降维... Xn -> F(xn) -> T(x) 机器学习－ (NYU,2002), Facebook AI, Google Deepmind Theano (University of Montreal, ~2010), 学院派 Kersa, “Deep Learning library for Theano and TensorFlow” Caffe (Berkeley)，卷积神经网络,贾扬清 TensorFlow (Google) Spark MLLib

footprint or quality, we should consider employing suitable efficient architectures. The progress of deep learning is characterized by the phases of architectural breakthroughs to improve on previous results and (CNNs) were another important breakthrough that enabled learning spatial features in the input. Recurrent Neural Nets (RNNs) facilitated learning from the sequences and temporal data. These breakthroughs optimization experience using these efficient layers and architectures. Let’s start our journey with learning about embeddings in the next section. Embeddings for Smaller and Faster Models We humans can intuitively

make it shorter.” Blaise Pascal In the last chapter, we discussed a few ideas to improve the deep learning efficiency. Now, we will elaborate on one of those ideas, the compression techniques. Compression gentle introduction to the idea of compression. Details of quantization and its applications in deep learning follow right after. The quantization section delves into the implementation details using code compression might lead to degradation in quality. In our case, we are concerned about compressing the deep learning models. What do we really mean by compressing though? As mentioned in chapter 1, we can break

them, with an eye towards conceptual understanding as well as practically using them in your deep learning models. We start with sparsity. If your goal was to optimize your brain for storage, you can different possible methods of picking the connections and nodes to prune, and how to prune a given deep learning model to achieve storage and latency gains with a minimal performance tradeoff. Next, the chapter effectiveness. Later on in this chapter, we have a project that relies on it for sparsifying a deep learning model. The authors of the Optimal Brain Damage (OBD) paper approximate the saliency score using

Machine Learning Lecture 10: Neural Networks and Deep Learning Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2018 Deep Feedforward usually a highly non-linear function • Feedforward networks are of extreme importance to machine learning practioners • The conventional neural networks (CNN) used for object recognition from photos are units), and output layer 7 / 19 Neural Feedforward Networks (Contd.) • We approximate f ∗(x) by learning f(x) from the given training data • In the output layer, f(x) ≈ y for each training data, but the