defining layers and neurons, enabling users to construct complex network architec- tures. • Backpropagation: The implementation of backpropa- gation in micrograd++ allows for efficient training of models following code snippet demonstrates the basic usage of micrograd++ for performing operations and backpropagation on scalar values. # include # include ” Value . hpp ” i n t main ( ) { auto a = multi-layer percep- tron, composed of multiple layers. It supports the training of models using backpropagation and gradient descent, allowing for the efficient optimization of network parameters. VI. SUPPORTING

the average over the loss functions from individual training samples L = 1 m m � i=1 Li • Backpropagation computes the gradients with respect to only one single training sample as given by ∂Li/∂w and ∂ ∂w[l] jk �� k w[l] jka[l−1] k + b[l] j � = a[l−1] k δ[l] j 17 / 19 Backpropagation Algorithm • The backpropagation equations provides us with a way of computing the gradient of the cost function

▪ New Issue: How to train MLP ▪ Chain Rules => Backpropagation http://www.iro.umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf Backpropagation: First Spark ▪ Derived in early 60’s ▪ Run on

move data to device (cpu/cuda) forward pass (compute output) compute loss compute gradient (backpropagation) update model with optimizer Neural Network Validation Loop model.eval() total_loss = 0

machine learning algorithms over the past two decades. Stochastic Gradient Descent (SGD) and Backpropagation were the well-known algorithms designed for training deep networks. However, one of the critical