layer and output layer) in the actual proposed neural network model. Keras provides a lot of pre-build layers so that any complex neural network can be easily created. Some of the important Keras layers Activation, Dense from keras import initializers my_init = initializers.VarianceScaling(scale=1.0, mode='fan_in', distribution='normal', seed=None) model.add(Dense(512, activation='relu', input_shape=(784 input_shape=(784,), kernel_initializer=my_init)) where,  scale represent the scaling factor  mode represent any one of fan_in, fan_out and fan_avg values  distribution represent either of normal or

h5py 快速开始 38 如 果 模 块 导 入 没 有 错 误， 那 么 模 块 已 经 安 装 成 功， 否 则 你 可 以 在 http://docs.h5py.org/en/latest/build.html 中找到详细的安装说明。 模型 39 4 模型 4.1 关于 Keras 模型 在 Keras 中有两类主要的模型：Sequential 顺序模型 和 使用函数式 API 的 Model 4.2.3 Sequential 模型方法 4.2.3.1 compile compile(self, optimizer, loss, metrics=None, sample_weight_mode=None, weighted_metrics=None, target_tensors=None) 用于配置训练模型。 参数 • optimizer: 字符串（优化器名）或者优化器对象。详见 {'output_a'：'accuracy'}。 • sample_weight_mode: 如果你需要执行按时间步采样权重（2D 权重） ，请将其设置为 temporal。默认为 None，为采样权重（1D）。如果模型有多个输出，则可以通过传递 mode 的字典或列表，以在每个输出上使用不同的 sample_weight_mode。 • weighted_metrics: 在训练和测试期间，由 sample_weight

tolerate approximate responses, since often there are no exact answers. Machine learning algorithms help build models, which as the name suggests is an approximate mathematical model of what outputs correspond Inference Efficiency By inference, we mean when the model is deployed and is in the prediction mode. Hence, inference efficiency primarily deals with questions that someone deploying a model would ask offline applications of these models. As an example, the Google Translate application supports offline mode which improves the user experience in low or no-connectivity areas. This is made possible with an

Out[20]: tensor(2., requires_grad=True) True 除了在创建时指定参数是否需要计算梯度，还可以随时改变张量的属性，调用张量的 requires_grad_(mode)函数即可，例如 w.requires_grad_(False)即可将待优化张量转变为普通 张量。 PyTorch 的自动梯度功能只允许对待优化张量进行梯度计算，例如： In [21]: label) # 打印这条句子的标签 # 构建词汇表，并分词编码，仅考虑 10000 个单词，耗时约 5 分钟 TEXT.build_vocab(train_data, max_size=10000, vectors='glove.6B.100d') LABEL.build_vocab(train_data) # 打印单词数量：10000++ print(f'Unique add(layers.ReLU())# 添加激活函数层 network.build(input_shape=(4, 4)) # 创建网络参数 network.summary() 上述代码通过指定任意的 layers_num 参数即可创建对应层数的网络结构，在完成网络创建 时，网络层类并没有创建内部权值张量等成员变量，此时通过调用类的 build 方法并指定 输入大小，即可自动创建所有层的内部张量。通过

ReLU Activation nn.ReLU() See here to learn about why we need activation functions. torch.nn – Build your own neural network import torch.nn as nn class MyModel(nn.Module): def __init__(self): return self.net(x) Initialize your model & define layers Compute output of your NN torch.nn – Build your own neural network import torch.nn as nn class MyModel(nn.Module): def __init__(self): criterion(pred, y) loss.backward() optimizer.step() iterate n_epochs set model to train mode iterate through the dataloader set gradient to zero move data to device (cpu/cuda) forward pass

TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) # This will print the output in the streaming mode. generated_ids = model.generate( model_inputs, max_new_tokens=512, streamer=streamer, ) 除了使用 TextStreamer output_dir: str, bias="none" ): """Collects the state dict and dump to disk.""" # check if zero3 mode enabled if deepspeed.is_deepspeed_zero3_enabled(): state_dict = trainer.model_wrapped._zero3_cons __init__() self.tokenizer = tokenizer self.max_len = max_len rank0_print("Formatting inputs...Skip in lazy mode") self.tokenizer = tokenizer self.raw_data = raw_data self.cached_data_dict = {} def __len__(self):

modules are scripted in amp. Disable autocast in TorchScript by using `torch._C._jit_set_autocast_mode(False)` if you encounter this issue. PyTorch RN-08516-001_v23.07 | 120 Chapter 19. PyTorch Release PyTorch RN-08516-001_v23.07 | 140 Known Issues ‣ Starting in 22.02 the PyTorch container does not build Caffe2 anymore. If scripted models were exported in the legacy format (using our 19.09 or previous heuristics, use export USE_HEURISTIC_MODE_B=1 in addition to export CUDNN_V8_API_ENABLED=1. Please refer to the cuDNN API docs for more information about this heuristic mode (https://docs.nvidia.com/deeplearning/cudnn/

• torch.no_grad() • Don’t store the history of all computations • eval() • Tell compiler which mode to run on. Visualization • TensorboardX (visualise training) • PyTorchViz (visualise computation again... Misc • Alternative : Static Computation Graphs: Alternative: Static graphs Step 1: Build computational graph describing our computation (including finding paths for backprop) Step 2:

with a better footprint? The next subsection elaborates it further. Using learning techniques to build smaller and faster efficient models Overall, as summarized in table 3-1, improving sample efficiency quality metrics and exchange those improvements to reduce footprint metrics. This was necessary to build an intuition of the real world problems they aim to tackle. Now, let’s dive into these learning techniques ModelCheckpoint( filepath=checkpoint_path, save_weights_only=True, monitor='val_categorical_accuracy', mode='max', save_best_only=True) def load_best_checkpoint(model, model_name): checkpoint_dir_path =

tradeoff on how much compression we want v/s how much quality loss can we tolerate? Let us slowly build up to that by exploring how quantization can help us. A Generic View of Quantization Quantization and so on). The model architecture is independent of the batch size. During inference (prediction mode), the typical value for the batch size is 1 because we predict one value at a time. The design of