transform large and complex models into smaller and efficient models capable of running on mobile and edge devices. We have also set up a couple of programming projects for a hands-on model optimization experience sentence “the quick brown fox jumps over the lazy dog”, we can mask the word “jumps” and let the neural network predict the word it thinks fits in the sentence based on the surrounding words (context). Mathematically step, we train a model which trains the embedding table along with it. We use a single hidden layer network9 with a softmax classification head for this task. The size of the softmax classification head is

models, the performance of the model scaled well with the number of labeled examples, since the network had a large number of parameters. Thus to extract the most out of the setup, the model needed a large in neural networks has led to an increase in the network complexity, number of parameters, the amount of training resources required to train the network, prediction latency, etc. Natural language models point values to 8-bit unsigned / signed integers). Quantization can generally be applied to any network which has a weight matrix. It can often help reduce the model size 2 - 8x, while also speeding up

Compression techniques are used to achieve an efficient representation of one or more layers in a neural network with a possible quality trade off. The efficiency goals could be the optimization of the model with techniques can help reduce complexity and improve generalization. Let us consider an arbitrary neural network layer. We can abstract it using a function with an input and parameters such that . In the case case of a fully-connected layer, is a 2-D matrix. Further, assume that we can train another network with far fewer parameters ( ) such that the outputs are approximately the same . Such a model is useful

High quality models have an additional benefit in footprint constrained environments like mobile and edge devices where they provide the flexibility to trade off some quality for smaller footprints. In the image in figure 3-6. Such a shift has two side-effects. First, a part of the image “falls off” the top edge. That information will be lost. And the second, the lower part of the image doesn’t have any pixel fed to improve the target models. 12 Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." arXiv preprint arXiv:1903.12261 (2019). 11

training process: performance and convergence. Hyperparameters like number of filters in a convolution network or 1 Note that this search space is just choosing if we are applying the techniques. The techniques extended beyond training parameters to structural parameters that can manipulate the structure of a network. The number of dense units, number of convolution channels or the size of convolution kernels can searched with the techniques that we discussed in this section. However, to truly design a Neural Network from scratch, we need a different approach. The next section dives into the search for neural architectures

实现更复杂的模型奠定了基础。接下来，在 6节 和 7节 中，我们介绍了卷积神经网络（convolutional neural network，CNN），这是构成大多数现代计算机视觉系统骨干的强大工具。随后，在 8节 和 9节 中，我们引入了循环神经网络(recurrent neural network，RNN)，这是一种利用数据中的时间或序列 结构的模型，通常用于自然语言处理和时间序列预测。在 10节 中，我们介绍了一类新的模型，它采用 ，甚至开发视频游戏的人工智能（AI）。深度强化学习（deep reinforcement learning）将深度学习应用于强化学习的问题，是非常热门的研究领域。突破性的深度Q网络 （Q‐network）在雅达利游戏中仅使用视觉输入就击败了人类，以及 AlphaGo 程序在棋盘游戏围棋中击败了世 界冠军，是两个突出强化学习的例子。 在强化学习问题中，智能体（agent）在一系列的时间步骤上 接的多层感知机中。因为这些网络特征元素的顺序是不变的，因此最优的结果是利用先验知识，即利用相近 像素之间的相互关联性，从图像数据中学习得到有效的模型。 本章介绍的卷积神经网络（convolutional neural network，CNN）是一类强大的、为处理图像数据而设计的 神经网络。基于卷积神经网络架构的模型在计算机视觉领域中已经占主导地位，当今几乎所有的图像识别、 目标检测或语义分割相关的学术竞赛和商业应用都以这种方法为基础。

Cython. Automatic differentiation is done with a tape-based system at both a functional and neural network layer level. This functionality brings a high level of flexibility and speed as a deep learning framework and provides accelerated NumPy-like functionality. PyTorch also includes standard defined neural network layers, deep learning optimizers, data loading utilities, and multi-gpu, and multi-node support. script is available on GitHub and NGC. ‣ Mask R-CNN model: Mask R-CNN is a convolution-based neural network that is used for object instance segmentation. PyTorch Release 23.07 PyTorch RN-08516-001_v23.07

weight of that connection). Can we do the same with neural networks? Can we optimally prune the network connections, remove extraneous nodes, etc. while retaining the model’s performance? In this chapter depicts two networks. The one on the left is the original network and the one on the right is its pruned version. Note that the pruned network has fewer nodes and some retained nodes have fewer connections Figure 5-1: An illustration of pruning weights (connections) and neurons (nodes) in a neural network consisting of fully connected layers. Exercise: Sparsity improves compression Let's import the

prepared for professionals who are aspiring to make a career in the field of deep learning and neural network framework. This tutorial is intended to make you comfortable in getting started with the Keras framework 12 Convolutional Neural Network (CNN) ........................................................................................................... 13 Recurrent Neural Network (RNN) .................. ........................................................... 71 12. Keras ― Convolution Neural Network ................................................................................................

模块的输出??连同它的网络层参数??和??等称为一层网络层。特别地，对于网络中间的 层，也叫作隐藏层，最后一层也叫作输出层。这种由大量神经元模型连接形成的网络结构 称为神经网络(Neural Network)。从这里可以看到，神经网络并不难理解，神经网络每层的 节点数和神经网络的层数或结构等决定了神经网络的复杂度。 预览版202112 第 3 章 分类问题 10 输入层：? 处理高维度的图片、视频数据时往往出现网络参数量巨大，训练非常困难的问题。通过利 用局部相关性和权值共享的思想，Yann Lecun 在 1986 年提出了卷积神经网络 (Convolutional Neural Network，简称 CNN)。随着深度学习的兴盛，卷积神经网络在计算机 视觉中的表现大大地超越了其它算法模型，呈现统治计算机视觉领域之势。这其中比较流 行的模型有用于图片分类的 AlexNet、VGG、Go 处理并理解文本数据是自然语言处理 的一个核心问题。卷积神经网络由于缺乏 Memory 机制和处理不定长序列信号的能力，并 不擅长序列信号的任务。循环神经网络(Recurrent Neural Network，简称 RNN)在 Yoshua Bengio、Jürgen Schmidhuber 等人的持续研究下，被证明非常擅长处理序列信号。1997 预览版202112 6.8 汽车油耗预测实战