+ ? ≥ 1 Negative class: ?!? + ? ≤ −1 ? = 1 ? Figure 2: Hard-margin SVM. aim of the above optimization problem is to find a hyperplane (parameterized by ω and b) with margin γ = 1/∥ω∥ maximized, while training set. 2.2 Preliminary Knowledge of Convex Optimization 2.2.1 Optimization Problems and Lagrangian Duality We now consider the following optimization problem min ω f(ω) (9) s.t. gi(ω) ≤ 0, i = 1 gk(ω) and the equality constraints h1(ω), · · · , hl(ω). We construct the Lagrangian of the above optimization problem as L(ω, α, β ) = f(ω) + k � i=1 αigi(ω) + l � j=1 β jhj(ω) (12) In fact, L(ω, α

Outline 1 SVM: A Primal Form 2 Convex Optimization Review 3 The Lagrange Dual Problem of SVM 4 SVM with Kernels 5 Soft-Margin SVM 6 Sequential Minimal Optimization (SMO) Algorithm Feng Li (SDU) SVM December 28, 2021 15 / 82 Convex Optimization Review Optimization Problem Lagrangian Duality KKT Conditions Convex Optimization S. Boyd and L. Vandenberghe, 2004. Convex Optimization. Cambridge university press press. Feng Li (SDU) SVM December 28, 2021 16 / 82 Optimization Problems Considering the following optimization problem min ω f (ω) s.t. gi(ω) ≤ 0, i = 1, · · · , k hj(ω) = 0, j = 1, · · · , l with

this using the earlier example for choosing quantization and/or clustering techniques for model optimization. We have a search space which has two boolean valued parameters: quantization and clustering hyperparameters. Some of the commonly tuned hyperparameters are the learning rate and the momentum of the optimization algorithm and the training batch size. Other aspects of the training pipeline like data augmentation may influence each other. Hence, we need a sophisticated approach to tune them. Hyperparameter Optimization (HPO) is the process of choosing values for hyperparameters that lead to an optimal model. HPO

Pytorch ● Dataset & Dataloader ● Tensors ● torch.nn: Models, Loss Functions ● torch.optim: Optimization ● Save/load models Prerequisites ● We assume you are already familiar with… 1. Python3 ■ deep neural networks Training Neural Networks Training Define Neural Network Loss Function Optimization Algorithm More info about the training process in last year's lecture video. Training & Testing calculation. Training & Testing Neural Networks – in Pytorch Define Neural Network Loss Function Optimization Algorithm Training Validation Testing Step 2. torch.nn.Module Load Data torch.nn – Network

the input model and wraps the prunable blocks for sparse training using TFMOT (Tensorflow Model Optimization) library. In this case, we prune the 50% of the weights in each prunable block using magnitude-based UpdatePruningStep() works in conjunction with the TFMOT pruning wrappers to update the wrappers after each optimization step. update_pruning = tfmot.sparsity.keras.UpdatePruningStep() callbacks = [update_pruning] centroids_init = np.linspace(x_sorted[0], x_sorted[-1], num_clusters) # Construct the variables in this optimization problem. # We will not update 'x', and hence it is not trainable. x_var = tf.Variable(initial_value=x_sorted

requires large computational resources, so they have to be carefully used. Automated Hyper-Param Optimization (HPO) is one such technique that can be used to replace / supplement manual tweaking of hyper-parameters allocate resources to promising ranges of hyper-parameters like Bayesian Optimization (Figure 1-12 illustrates Bayesian Optimization). These algorithms construct ‘trials’ of hyper-parameters, where each trial across them is how future trials are constructed based on past results. Figure 1-12: Bayesian Optimization over two dimensions x1 and x2. Red contour lines denote a high loss value, and blue contour lines

Keras is an optimal choice for deep learning applications. Features Keras leverages various optimization techniques to make high level neural network API easier and more performant. It supports the Optimizer are used in learning phase to find the error (deviation from actual output) and do optimization so that the error will be minimized.  Fit the model: The actual learning process will optimize the layer (and the model) by dynamically applying the penalties on the weights during optimization process. To summarise, Keras layer requires below minimum details to create a complete layer

• 变量数目非常庞大 • 内存空间需求大 • 计算耗时 • 迭代的局部集束调整 • 大误差难以均匀扩散到整个序列 • 极易陷入局部最优 • 姿态图优化（Pose Graph Optimization） • 只优化相机之间的相对姿态，三维点都消元掉； • 是集束调整的一个近似，不是最优解。 基于自适应分段的集束调整 • 将长序列分成若干段短序列； • 每个短序列进行独立的Sf Recognition Pose Graph Optimization + Traditional BA Street序列结果比较 ENFT-SLAM ORB-SLAM Non-consecutive Track Matching Segment-based BA Bag-of-words Place Recognition Pose Graph Optimization + Traditional BA

|✓T x(i) � y(i)|. 2 Gradient Descent Gradient Descent (GD) method is a first-order iterative optimization algorithm for finding the minimum of a function. If the multi-variable function J(✓) is di↵erentiable known as incremental gradient descent, is a stochastic approximation of the gradient descent optimization method. In each iteration, the parameters are updated according to the gra- dient of the error

[6] Stephen Boyd, Lieven Vandenberghe, Convex Optimization[M]. Cambridge: Cambridge University Press, 2004. [7] PLATT J. Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector