PyTorch Tutorial Willie Chang Pranay Manocha Installing PyTorch • ???????????? On your own computer • Anaconda/Miniconda: conda install pytorch -c pytorch • Others via pip: pip3 install torch • ??

Keras i Keras ii About the Tutorial Keras is an open source deep learning framework for python. It has been developed by an artificial intelligence researcher Leading organizations like Google, Square, Netflix, Huawei and Uber are currently using Keras. This tutorial walks through the installation of Keras, basics of deep learning, Keras models, Keras layers, applications. Audience This tutorial is prepared for professionals who are aspiring to make a career in the field of deep learning and neural network framework. This tutorial is intended to make you comfortable

Machine Learning Pytorch Tutorial TA : 曾元（Yuan Tseng） 2022.02.18 Outline ● Background: Prerequisites & What is Pytorch? ● Training & Testing Neural Networks in Pytorch ● Dataset & Dataloader ● Tensors implementations of recent deep learning papers ○ ... References ● Machine Learning 2021 Spring Pytorch Tutorial ● Official Pytorch Tutorials ● https://numpy.org/ Any questions?

model: This model was introduced in the Aggregated Residual Transformations for Deep Neural Networks paper. It is based on the regular ResNet model, which substitutes 3x3 convolutions in the bottleneck block added Squeeze-and- Excitation (SE) module that was introduced in the Squeeze-and-Excitation Networks paper. This model script is available on GitHub. ‣ TransformerXL model: This transformer-based language Our implementation is based on the codebase that was published by the authors of the Transformer-XL paper. Our implementation uses modified model architecture hyperparameters, our modifications were made

compute during the pre-training phase and pre-training BERT-like models is not cheap. The original paper reports BERT-Base requiring 4 Cloud TPU Pods (4 chips each, total 16 chips) over 4 days for a total the loss to be minimized is a variant of the cross-entropy loss. We would refer you to the SimCLR paper for more details about the chosen loss functions and other alternatives considered. Once the desired presented in the context of different model architectures and hyperparameters. For example, the paper titled: ResNet Strikes Back by Wightman et al.14 demonstrates improvement in the accuracy achieved

Machine: 1992 http://www.iro.umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf Dark time ▪ Paper got rejected ▪ Hinton moved to CIFAR seeking for funding ▪ Conspiracy: rebrand“neural network”as ctures/backprop_old.pdf Deep is more efficient: representation learning http://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks.pdf Another Hero: NVIDIA http://www.iro.umontreal 60,000,000 parameters ▪ 5 conv layers ▪ 26.2% -> 15.3%, top-5 error rate https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional- neural-networks.pdf http://www.iro.umontreal

architectures. HPO requires all the hyperparameters to be known prior to the start of the search. In their paper titled "Neural Architecture Search With Reinforcement Learning"5, Zoph et. al. employed neural networks large controller is required which would invariably lead to higher search expenses. In a follow up paper, Zoph et. al. addressed the above shortcomings with a novel controller architecture called NASNet6 predict the design of two cells, the total number of predicted parameters is . In the original NASNet paper, the value for is chosen to be 5. Figure 7-8 (right) shows a predicted block. Figure 7-8: The structure

daunting? Don’t fret! Let’s start with how to compute the saliency scores. Saliency Scores In the paper Optimal Brain Damage1, LeCun et al. suggested that as much as 50% of the connections (weights) from relies on it for sparsifying a deep learning model. The authors of the Optimal Brain Damage (OBD) paper approximate the saliency score using a second-derivative of the weights , where is the loss function growth is higher in layers which have a higher impact on the loss value. Han et al. in their seminal paper titled "Learning both Weights and Connections for Efficient Neural Networks8" proposed a three step

used for learning and scientific research and is freely disseminated, but it must not be used for commercial purposes. Otherwise, the contributor is not responsible for the consequences. 目录 I 目录 1 Keras: True，则网络将展开，否则将使用符号循环。展开可以 加速 RNN，但它往往会占用更多的内存。展开只适用于短序列。 参考文献 • Long short-term memory (original 1997 paper) • Learning to forget: Continual prediction with LSTM • Supervised sequence labeling with recurrent

BREIMAN L. Random forests[J]. Machine learning, 2001, 45(1): 5–32. [5] Ridgeway G . Special Invited Paper. Additive Logistic Regression: A Statistical View of Boosting: Discussion[J]. Annals of Statistics