” , 7: ” Sneaker ” , 8: ”Bag” , 9: ”Ankle␣Boot” , } import matplotlib . pyplot as plt f i g u r e = plt . f i g u r e () # 抽 取 索 引 为 100 的 数 据 来 显 示 img , l a b e l = training_data [ 1 0 0 ] plt 迭 代 取 出 的 数 据 量 # s h u f f l e ： 洗 牌 的 意 思， 先 把 数 据 打 乱， 然 后 再 分 为 不 同 的 batch Chapter 1. 准备章节 9 train_dataloader = DataLoader ( training_data , batch_size =64, s h u f f l e= True ) test_dataloader h u f f l e=True ) 我们写点程序检测一下 DataLoader： train_features , train_labels = next ( i t e r ( train_dataloader ) ) print ( f ” Feature ␣batch␣shape : ␣{ train_features . s i z e () }” ) print ( f ” Labels

import torchvision from PIL import Image from torch import nn from torch.nn import functional as F from torch.utils import data from torchvision import transforms 目标受众 本书面向学生（本科生或研究生）、工程师和研究人员，他 Ra×b: 包含a行和b列的实数矩阵集合 • A ∪ B: 集合A和B的并集 13 • A ∩ B：集合A和B的交集 • A \ B：集合A与集合B相减，B关于A的相对补集 函数和运算符 • f(·)：函数 • log(·)：自然对数 • exp(·): 指数函数 • 1X : 指示函数 • (·)⊤: 向量或矩阵的转置 • X−1: 矩阵的逆 • ⊙: 按元素相乘 • [·, ∇xy：y关于x的梯度 • � b a f(x) dx: f在a到b区间上关于x的定积分 • � f(x) dx: f关于x的不定积分 14 目录 概率与信息论 • P(·)：概率分布 • z ∼ P: 随机变量z具有概率分布P • P(X | Y )：X | Y 的条件概率 • p(x): 概率密度函数 • Ex[f(x)]: 函数f对x的数学期望 • X ⊥ Y : 随机变量X和Y

function (PDF) f (x, y) P(a1 ≤ X ≤ b1, a2 ≤ Y ≤ b2) = � b1 a1 � b2 a2 f (x, y)dxdy Marginal probability density functions fX(x) = � ∞ −∞ f (x, y)dy for − ∞ < x < ∞ fY (x) = � ∞ −∞ f (x, y)dx for to more than two random variables P(a1 ≤ X1 ≤ b1, · · · , an ≤ Xn ≤ bn) = � b1 a1 · · · � bn an f (x1, · · · , xn)dx1 · · · dxn Feng Li (SDU) GDA, NB and EM September 27, 2023 12 / 122 Independent and Y Joint PDF f (x, y) Marginal PDF fX(x) = � y f (x, y)dy The Conditional probability density function of Y given X = x fY |X(y | x) = f (x, y) fX(x) , ∀y or fY |X=x(y) = f (x, y) fX(x) , ∀y

���� LFW�Labeled Faces in the Wild�����������������99.63%� ������������������������������ Schroff, F., Kalenichenko, D. and Philbin, J., 2015. Facenet: A unified embedding for face recognition and clustering �t������i 6000 �����h�����vh�i 300 �u��300 ���� ���ha��c��d����t���LFW���s�d�����p� 2013�:�����������f�������l��+�c��� 2014�:����������c��� 2014��s��c��+���.����tw����/e������������� ������������� • CASIA-WebFace ������������� mke������F������g�������r��AF���� �b���mkeC������C��b���������S���i������W� cn���mke��l�h�����������K��mkeC������w ����cntp_�������f�d�as�������I_cn����� ����� �������

Derivative: The directional derivative of function f : Rn → R in the direction u ∈ Rn is ∇uf (x) = lim h→0 f (x + hu) − f (x) h ∇uf (x) represents the rate at which f is increased in direction u When u is the the i-th standard unit vector ei, ∇uf (x) = f ′ i (x) where f ′ i (x) = ∂f (x) ∂xi is the partial derivative of f (x) w.r.t. xi Feng Li (SDU) Linear Regression September 13, 2023 10 / 31 Gradient (Contd derivative of f in the direction of u can be represented as ∇uf (x) = n � i=1 f ′ i (x) · ui Feng Li (SDU) Linear Regression September 13, 2023 11 / 31 Gradient (Contd.) Proof. Letting g(h) = f (x + hu)

problem min ω f (ω) s.t. gi(ω) ≤ 0, i = 1, · · · , k hj(ω) = 0, j = 1, · · · , l with variable ω ∈ Rn, domain D = �k i=1 domgi ∩�l j=1 domhj, optimal value p∗ Objective function f (ω) k inequality 2021 17 / 82 Lagrangian Lagrangian: L : Rn × Rk × Rl → R, with domL = D × Rk × Rl L(ω, α, β ) = f (ω) + k � i=1 αigi(ω) + l� j=1 β jhj(ω) Weighted sum of objective and constraint functions αi is Function The Lagrange dual function G : Rk × Rl → R G(α, β ) = inf ω∈D L(ω, α, β ) = inf ω∈D � �f (ω) + k � i=1 αigi(ω) + l� j=1 β jhj(ω) � � G is concave, can be −∞ for some α, β Feng Li (SDU)

0a0+4136153 TensorRT 8.6.1.6 23.05 22.04 NVIDIA CUDA 12.1.1 2.0.0 TensorRT 8.6.1.2 23.04 2.1.0a0+fe05266f TensorRT 8.6.1 23.03 NVIDIA CUDA 12.1.0 2.0.0a0+1767026 23.02 TensorRT 8.5.3 23.01 NVIDIA CUDA 0a0+410ce96 22.11 1.13.0a0+936e930 TensorRT 8.5.1 22.10 22.09 NVIDIA CUDA 11.8.0 1.13.0a0+d0d6b1f TensorRT 8.5.0.12 22.08 NVIDIA CUDA 11.7.1 1.13.0a0+d321be6 TensorRT 8.4.2.4 22.07 1.13.0a0+08820cb TensorRT 7.2.2.3+cuda11.1.0.024 20.12 NVIDIA CUDA 11.1.1 1.8.0a0+1606899 TensorRT 7.2.2 20.11 1.8.0a0+17f8c32 20.10 NVIDIA CUDA 11.1.0 1.7.0a0+7036e91 TensorRT 7.2.1 20.09 1.7.0a0+8deb4fe 20.08 NVIDIA

function f : Rn → R is convex, if domf is a convex set and if for all x, y ∈ domf and λ with 0 ≤ λ ≤ 1, we have f(λx + (1 − λ)y) ≤ λf(x) + (1 − λ)f(y) If f(λx + (1 − λ)y) ≥ λf(x) + (1 − λ)f(y) f is said Inequality Theorem 1. Jensen’s Inequality Let f(x) be a convex function defined on an interval I. If x1, x2, · · · , xN ∈ I and λ1, λ2, · · · , λN ≥ 0 with �N i=1 λi = 1 f( N � i=1 λixi) ≤ N � i=1 λif(xi) N = 2, we have f(λ1x1 + λ2x2) ≤ λ1f(x1) + λ2f(x2) due to convexity of f(x). When N ≥ 3, the proof is by induction. We assume that, the Jensen’s inequality holds when N = k − 1, i.e. f( k−1 � i=1 λixi)

or ( verbose == 1 and ((step_idx + 1) % ((int)(num_steps / 5)) == 0)): print(f'Step: {step_idx + 1}, Loss: {loss:.5f}.') # Compute the gradients w.r.t. only the centroids_var. gradients = tape.gradient(loss reconstruction_error = compute_reconstruction_error(x, decoded_x) vprint(verbose, f'Final reconstruction error: {reconstruction_error:.4f}.') return decoded_x, computed_centroids, reconstruction_error In order simulate_clustering( img, num_clusters, num_steps=10, learning_rate=2e-1, verbose=2) print(f'Reconstruction error: {reconstruction_error:.4f}.') plt.axis('off') plt.imshow(decoded_img) Let us try clustering with 22

Optimization Problems and Lagrangian Duality We now consider the following optimization problem min ω f(ω) (9) s.t. gi(ω) ≤ 0, i = 1, · · · , k (10) hj(ω) = 0, j = 1, · · · , l (11) where ω ∈ D is the the constraints. The aim of the above optimiza- tion problem is to minimizing the objective function f(ω) subject to the inequal- ity constraints g1(ω), · · · , gk(ω) and the equality constraints h1(ω), · · · , hl(ω). We construct the Lagrangian of the above optimization problem as L(ω, α, β ) = f(ω) + k � i=1 αigi(ω) + l � j=1 β jhj(ω) (12) In fact, L(ω, α, β ) can be treated as a weighted sum