and Y are discrete random variable Joint probability mass function (PMF) pX,Y (x, y) = P(X = x, Y = y) Marginal probability mass function for discrete random variables pX(x) = � y P(X = x, Y = y) = = � y P(X = x | Y = y)P(Y = y) pY (y) = � x P(X = x, Y = y) = � x P(Y = y | X = x)P(Y = x) Extension to multiple random variables X1, X2, X3, · · · , Xn pX(x1, x2, · · · , xn) = P(X1 = x1, X2 = x2 distribution Suppose both X and Y are continuous random variable Joint probability density function (PDF) f (x, y) P(a1 ≤ X ≤ b1, a2 ≤ Y ≤ b2) = � b1 a1 � b2 a2 f (x, y)dxdy Marginal probability density

and Y ∈ {0, 1} is a random variable representing if there is a cat in the given image. Now, given an image x = [x1, x2, · · · , xn]T , out goal is to calculate P(Y = y | X = x) = P(X = x | Y = y)P(Y = = y) P(X = x) (2) where y ∈ {0, 1}. In particular, P(Y = y | X = x) is the probability that the image is labeled by y given that the image can be represented by feature vector x, P(X = x | Y = y) is given that it is labeled by y, P(Y = y) is the probability that a randomly picked image is labeled by y, and P(X = x) is the probability that a randomly picked image has label y. In our case, we make decision

Rn according to sign(ωT x + b). Specifically, given a point x0 ∈ Rn, its label y is defined as y0 = sign(ωT x0 + b), i.e. y0 = � 1, ωT x0 + b ≥ 0 −1, otherwise (2) Given any x0 ∈ Rn, we can calculate sign of the distance, i.e., sign(γ), can be indicated by y0 = sign(ωT x0 + b). Therefore, we define the (unsigned) geometric distance of x0 as γ0 = y0(ωT x0 + b) ∥ω∥ (4) γ0 is the so-called margin of x0 data {(x(i), y(i))}i=1,··· ,m, we first assume that they are linearly separable. Specifically, there exists a hyperplane (parameterized by ω and b) such that ωT x(i) + b ≥ 0 for ∀i with y(i) = 1, while

into the Lagrangian to get the Lagrange dual function as follows L(ω, b, α) = 1 2∥ω∥2 − m � i=1 αi[y(i)(ωT x(i) + b) − 1] = 1 2ωT ω − m � i=1 αiy(i)ωT x(i) − m � i=1 αiy(i)b + m � i=1 αi = 1 2ωT j=1 αiαjy(i)y(j)(x(i))T x(j) − b m � i=1 αiy(i) + m � i=1 αi = m � i=1 αi − 1 2 m � i=1,j=1 αiαjy(i)y(j)(x(i))T x(j) − b m � i=1 αiy(i) = m � i=1 αi − 1 2 m � i=1,j=1 αiαjy(i)y(j)(x(i))T i=1 αi − 1 2 m � i=1,j=1 αiαjy(i)y(j)(x(i))T x(j) = m � i=1 αi − 1 2 m � i=1,j=1 αiαjy(i)y(j) < x(i), x(j) > (3) 2 Corollaries on Page 34 If αi = 0, y(i)(ωT x(i) + b) ≥ 1 ∵ αi = 0, αi + ri

A hyperplane based linear classifier defined by ω and b Prediction rule: y = sign(ωTx + b) Given: Training data {(x(i), y(i))}i=1,··· ,m Goal: Learn ω and b that achieve the maximum margin For now, signed If y (i) = 1, γ(i) ≥ 0; otherwise, γ(i) < 0 !" !" !"# + % = 0 ! !(#) !(#) = & & ' ((#) + * & Feng Li (SDU) SVM December 28, 2021 6 / 82 Margin (Contd.) Geometric margin γ(i) = y(i) �� ω )(#) + + ' Feng Li (SDU) SVM December 28, 2021 7 / 82 Margin (Contd.) Geometric margin γ(i) = y(i) �� ω ∥ω∥ �T x(i) + b ∥ω∥ � Scaling (ω, b) does not change γ(i) !" !" !"# + % = 0 ! !(#)

面的命令创建一个新的环境： 8 https://conda.io/en/latest/miniconda.html 9 conda create --name d2l python=3.9 -y 现在激活 d2l 环境： conda activate d2l 安装深度学习框架和d2l软件包 在安装深度学习框架之前，请先检查计算机上是否有可用的GPU。例如可以查看计算机是否装有NVIDIA • |X|：集合的基数 • ∥ · ∥p: ：Lp 正则 • ∥ · ∥: L2 正则 • ⟨x, y⟩：向量x和y的点积 • �: 连加 • �: 连乘 • def =：定义 微积分 • dy dx：y关于x的导数 • ∂y ∂x：y关于x的偏导数 • ∇xy：y关于x的梯度 • � b a f(x) dx: f在a到b区间上关于x的定积分 • � f(x) dx: P(X | Y )：X | Y 的条件概率 • p(x): 概率密度函数 • Ex[f(x)]: 函数f对x的数学期望 • X ⊥ Y : 随机变量X和Y 是独立的 • X ⊥ Y | Z: 随机变量X和Y 在给定随机变量Z的条件下是独立的 • Var(X): 随机变量X的方差 • σX: 随机变量X的标准差 • Cov(X, Y ): 随机变量X和Y 的协方差 • ρ(X, Y ): 随机变量X和Y

现在，你可以批量地在训练数据上进行迭代了： # x_train 和 y_train 是 Numpy 数组 -- 就像在 Scikit-Learn API 中一样。 model.fit(x_train, y_train, epochs=5, batch_size=32) 或者，你可以手动地将批次的数据提供给模型： model.train_on_batch(x_batch, y_batch) 只需一行代码就能评估模型性能： 只需一行代码就能评估模型性能： loss_and_metrics = model.evaluate(x_test, y_test, batch_size=128) 或者对新的数据生成预测： classes = model.predict(x_test, batch_size=128) 构建一个问答系统，一个图像分类模型，一个神经图灵机，或者其他的任何模型，就是这么 的快。深度学习背后的思想很简单，那么它们的实现又何必要那么痛苦呢？ compile(optimizer='rmsprop', loss='mse') # 自定义评估标准函数 import keras.backend as K def mean_pred(y_true, y_pred): return K.mean(y_pred) model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'

given an input feature vector. We assume a n-dimensional feature vector is denoted by x 2 Rn, while y 2 R is the output variable. In linear regression models, the hypothesis function is defined by h✓(x) so-called training. The training procedure is performed based on a given set of m training data {x(i), y(i)}i=1,··· ,m. In particular, we are supposed to find a hypothesis function (parameterized by ✓) which as follows J(✓) = 1 2 m X i=1 ⇣ h✓(x(i)) � y(i)⌘2 Our linear regression problem can be formulated as min ✓ J(✓) = 1 2 m X i=1 ⇣ ✓T x(i) � y(i)⌘2 1 Figure 1: 3D linear regression. Specifically

Tumor: Malignant/Benign? The classification result can be represented by a binary variable y ∈ {0, 1} y = � 0 : “Negative Class” (e.g., benign tumor) 1 : “Positive Class” (e.g., malignant tumor) Feng ? = ?"? 0.5 The threshold classifier output hθ(x) at 0.5 If hθ(x) ≥ 0.5, predict y = 1 If hθ(x) < 0.5, predict y = 0 Feng Li (SDU) Logistic Regression September 20, 2023 4 / 29 Warm-Up (Contd.) When new training example comes Tumor Size Malignant? (Yes) 1 (No) 0 0.5 An interesting observation y ∈ {0, 1} in classification problem, but the linear regression model hθ(x) = θTx can be > 1 or < 0 to

Target: output variable, y; Training example: (x(i), y(i)), i = 1, 2, 3, ..., m Hypothesis: h : X → Y. Training set house.) (living area of Learning algorithm h predicted y x (predicted price) 2023 5 / 31 Linear Regression (Contd.) Input: Training set (x(i), y(i)) ∈ R2 (i = 1, ..., m) Goal: Model the relationship between x and y such that we can predict the corresponding target according to a Linear Regression September 13, 2023 6 / 31 Linear Regression (Contd.) The relationship between x and y is modeled as a linear function. The linear function in the 2D plane is a straight line. Hypothesis: