nn.Module 主讲人：龙良曲 Magic ▪ Every Layer is nn.Module ▪ nn.Linear ▪ nn.BatchNorm2d ▪ nn.Conv2d ▪ nn.Module nested in nn.Module 1. embed current layers ▪ Linear ▪ ReLU ▪ Sigmoid ▪ Conv2d ▪ ConvTransposed2d

MF (Intel 80186) 1990 10 K （光学字符识别） 10 MB 10 MF (Intel 80486) 2000 10 M （网页） 100 MB 1 GF (Intel Core) 2010 10 G （广告） 1 GB 1 TF (Nvidia C2050) 2020 1 T （社交网络） 100 GB 1 PF (Nvidia DGX‐2) 很明显，随机存取存储 Torch24和Theano25。许多开创性的论文都是用这些工具写的。到目前为止，它们已经被TensorFlow26 （通常通过其高级API Keras27使用）、CNTK28、Caffe 229和Apache MXNet30所取代。第三代工具，即用 于深度学习的命令式工具，可以说是由Chainer31率先推出的，它使用类似于Python NumPy的语法来 描述模型。这个想法被PyTorch32、MXNet的Gluon com/caffe2/caffe2 30 https://github.com/apache/incubator‐mxnet 31 https://github.com/chainer/chainer 32 https://github.com/pytorch/pytorch 33 https://github.com/apache/incubator‐mxnet 34 https://github

cuBLAS 12.1.3.1 ‣ NVIDIA cuDNN 8.9.3 ‣ NVIDIA NCCL 2.18.3 ‣ NVIDIA RAPIDS™ 23.06 ‣ Apex ‣ rdma-core 39.0 ‣ NVIDIA HPC-X 2.15 ‣ OpenMPI 4.1.4+ ‣ GDRCopy 2.3 ‣ TensorBoard 2.9.0 ‣ Nsight Compute For more information about AMP, see the Training With Mixed Precision Guide. Tensor Core Examples The tensor core examples provided in GitHub and NGC focus on achieving the best performance and convergence GitHub. ‣ SE-ResNext model: This ResNeXt101-32x4d model has an added Squeeze-and- Excitation (SE) module that was introduced in the Squeeze-and-Excitation Networks paper. This model script is available

............................................................................................ 18 Core Modules ......................................................................................... ... 20 backend module ............................................................................................................................................ 21 utils module ................. intelligence(AI), audio & video recognition and image recognition. Artificial neural network is the core of deep learning methodologies. Deep learning is supported by various libraries such as Theano, TensorFlow

attention_mask=ret["attention_mask"][0], ) self.cached_data_dict[i] = ret return ret def make_supervised_data_module( tokenizer: transformers.PreTrainedTokenizer, data_args, max_len, ) -> Dict: """Make dataset and return dict(train_dataset=train_dataset, eval_dataset=eval_dataset) 然 后 我 们 利 用 make_supervised_data_module ， 通 过 使 用 SupervisedDataset 或 LazySupervisedDataset 来构建数据集。 def train(): global local_rank parser enable_input_require_grads() (续下页) 1.12. 有监督微调 33 Qwen (接上页) # Load data data_module = make_supervised_data_module( tokenizer=tokenizer, data_args=data_args, max_len=training_args.model_max_ �→length

establish our motivation behind seeking efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient tradeoff. To that end, we can think of work on efficient deep learning to be categorized in roughly four core areas, with infrastructure and hardware forming the foundation (see Figure 1-7). 7 Lossy compression detail in Chapter 2. (Figure 1-7: A mental model of Efficient Deep Learning, which comprises the core areas and relevant techniques as well as the foundation of infrastructure, hardware and tools.)

数据 程序模型 GPUs & 计算加速 深度学习的爆发 图像理解 自然语言处理 语音识别 机器自主 AWS 之上的人工智能应用 Zillow • Zestimate (using Apache Spark) Howard Hughes Corp • Lead scoring for luxury real estate purchase predictions FINRA • Anomaly frame/sec with 640x480 resolution 处处可部署 Beyond BlindTool by Joseph Paul Cohen, demo on Nexus 4 Fit the core library with all dependencies into a single C++ source file Easy to compile on

ropy', optimizer='rmsprop') 关于 KERAS 网络层 117 5.13 编写你自己的 Keras 层 对于简单、无状态的自定义操作，你也许可以通过 layers.core.Lambda 层来实现。但是 对于那些包含了可训练权重的自定义层，你应该自己实现这种层。 这是一个 Keras2.0 中，Keras 层的骨架（如果你用的是旧的版本，请你更新）。你只需要实 Keras Model 对象。 参考文献 • Rethinking the Inception Architecture for Computer Vision License 预训练权值基于 Apache License。 13.3.6 InceptionResNetV2 keras.applications.inception_resnet_v2.InceptionResNetV2(include_top=True Keras Model 对象。 参考文献 • Rethinking the Inception Architecture for Computer Vision License 预训练权值基于 Apache License。 13.3.7 MobileNet keras.applications.mobilenet.MobileNet(input_shape=None, alpha=1.0,

我们将网络实现为一个自定义网络类，只需要在初始化函数中创建各个子网络层，并 在前向计算函数 forward 中实现自定义网络类的计算逻辑即可。自定义网络类继承自 nn.Module 基类，这也是自定义网络类的标准写法，以方便地利用 nn.Module 基类提供的 parameters、load_state_dict、state_dict 等各种便捷功能。网络模型类实现如下： import torch nn from torch.nn import functional as F from torch import optim class MyNetwork(nn.Module): def __init__(self): super(MyNetwork, self).__init__() # 创建 3 个全连接层 datasets from torchvision import transforms from torch import nn, optim class LeNet5(nn.Module): def __init__(self): super(LeNet5, self).__init__() # 网络容器

计算任务到指定GPU设备 设备亲和性影响较小 设备亲和性影响较大 TensorFlow on Yarn技术细节揭秘 Yarn支持GPU调度ResourceManager端实现：� 扩展org.apache.hadoop.yarn.api.records.Resource抽象类及其实现，增加：� � public abstract int getGpuCores();� � public abstract � 1、对NodeManager GPU卡数量的统计管理� 2、调度器统计管理每个Pool的GPU设备数的分配情况� � 具体可以参考下面Patch的实现思路：� https://issues.apache.org/jira/browse/YARN-5517� TensorFlow on Yarn技术细节揭秘 Yarn支持GPU调度NodeManager端实现：� NodeManager yarn-site