Aller au boulot

Definition

Scalar $\alpha \in \mathbf{R}$,

Vector $x\in {\mathbf{R}}^n,n\in \mathbf{N}$

Linearity $f:{\mathbf{R}}^n\mapsto {\mathbf{R}}^m$,

$$f(\alpha \cdot x+\beta \cdot y)=\alpha \cdot f(x)+\beta \cdot f(y)$$

Estimator

Bias

$$\mathrm{bias}(\hat{y})=E(\hat{y})-y$$

$\hat{y}$ unbiased if $\mathrm{bias}(\hat{y})=0$

Mean Squared Error

$$MSE=E((\hat{y}-y{)}^2)=\mathrm{bias}(\hat{y}{)}^2+\mathrm{Var}(\hat{y})$$

minimizing the mean squared error = maximum likelihood estimator

Moore-Penrose pseudoinverse

$$X^{+}=\underset{\alpha \to 0}{\lim }(X^{T}X+\alpha I{)}^{-1}{X}^T$$

Probability

$P(y,x)$ represent the probability of $x$ and $y$ hanppend.

$P(y|x)$ represent the probability of $y$ while $x$ already known been happend.

Conditional Probability

$$P(y|x)=\frac{P(y,x)}{P(x)}$$

If $x$ independent with $y$, then

$$P(y|x)=P(y),P(y|x)=P(y,x)$$

Bayes's Rule

$$P(x|y)=\frac{P(x)P(y|x)}{P(y)}$$

Experience

$$E(x_i)=\sum _{i}P(x_i)x_i$$

$$E(x)={\int }_{I}P(x)xdx$$

Variance

$$\mathrm{Var}(x)=E((x-E(x){)}^2)$$

Covariance

$$\mathrm{Cov}(x,y)=E((x-E(x))(y-E(y)))$$

Normal Distribution

$$\mathcal{N}(\mu ,{\sigma }^2)=\sqrt{\frac{1}{2\pi {\sigma }^2}}{e}^{-\frac{1}{2{\sigma }^2}(\cdot -\mu {)}^2}$$

If $x\sim \mathcal{N}(\mu ,{\sigma }^2)$, then $E(x)=\mu$, $\mathrm{Var}(x)={\sigma }^2$.

Logistic sigmoid

$$s(x)=\frac{1}{1+e^{-x}}$$

Softplus function

$$\zeta (x)=\log (1+e^x)$$

Softmax

$$\sigma (x{)}_i=\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}$$

$L^p$ norm

$$||x|{|}_p={\left(\sum _i|x_i{|}^p\right)}^{\frac{1}{p}}$$

XOR

Linear model cannot perform XOR operation

$$f(x,W,c,w,b)=w^T\cdot max(0,W^{T}x+c)+b$$

$$f\left(\left[\begin{array}{cc}0& 0\\ 0& 1\\ 1& 0\\ 1& 1\end{array}\right]\right)=\left[\begin{array}{c}0\\ 1\\ 1\\ 0\end{array}\right]$$

$$W=\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right],c=\left[\begin{array}{c}0\\ -1\end{array}\right],w=\left[\begin{array}{c}1\\ -2\end{array}\right],b=0$$

General Problem

For a data-set with $N$ samples

$$(x_i,y_i)\in {\mathbf{R}}^n\times {\mathbf{R}}^m,i\in \mathbf{N},$$

$$f:{\mathbf{R}}^n\mapsto {\mathbf{R}}^m$$

$\hat{y}=f(x)$

$$||\hat{y}-y|{|}_2^2=\sum _i({\hat{y}}_i-y_i{)}^2$$

Linear Case

Module $x\in {\mathbf{R}}^n,$ $w\in {\mathbf{R}}^n\times {\mathbf{R}}^m,$ $y\in {\mathbf{R}}^m,$ $b\in {\mathbf{R}}^m,$

$$\hat{y}=f(x)=x\cdot w+b$$

Likelihood function

Likelihood is the probability that a particular outcome $x$ is observed when the true value of the parameter is $\theta$,

$$\mathcal{L}(\theta \mid x)=p_{\theta }(x)=P_{\theta }(X=x)$$

Unlike probabilities, likelihood function do not have to integrate (or sum) to 1.

Quadratic Problem

$$F(x)=\frac{1}{2}xw{x}^T-bx$$

$$\nabla F(x)=xw-b$$

$${\nabla }^{2}F(x)=w$$

$w$ is invertable,

$$x_1=x_0-\nabla F(x_0)({\nabla }^{2}F(x_0){)}^{-1}=x_0-(x_{0}w-b)w^{-1}=b{w}^{-1}=x^{\ast }$$

$$\hat{w}=\underset{w}{\mathrm{argmin}}S(w)$$

summary

• maximum likelihood estimation
• least square regression
• minimum cross-entropy between the distributions
• minimum the KL divergence

prevent overfitting

• maximum a posteriori estimation
• regularized least square

cross-entropy $\ne$ negative log-likelihood of a Bernoulli or softmax distribution

PCA

• maximize variance of data after projection

maximize variance -- lagrangian multiplier --> eigenvector of covariance matrix

Theorem (Bochner’s theorem)

A continuous function of the form 𝑘(𝑥,𝑦)=𝑘(𝑥−𝑦) is positive definite if and only if 𝑘(𝛿) is the Fourier transform of a non-negative measure.

Random Fourier features — Random walks