Aller au boulot

Regression

Linear Regression

Objective function

$$\underset{w}{\min }(\Vert y-x\cdot w{\Vert }_2^2)$$

Ordinary least square Lagrangian function

$$L(w)=||y-x\cdot w|{|}_2^2=(y-x\cdot w{)}^T(y-x\cdot w)=y^{T}y-w^T{x}^{T}y-y^{T}xw+w^T{x}^{T}xw$$

$$\frac{\partial L(w)}{\partial w}=2x^{T}xw-2x^{T}y$$

Let $\partial L(w)/\partial w=0$ leads to

$$w=(x^{T}x{)}^{-1}{x}^{T}y$$

With respect to different constraints,

• $||w|{|}_1<t$ for lasso,
• $||w|{|}_2<t$ for ridge.

Ridge Regression : L1 Regularization

Least absolute shrinkage and selection operator

$$\underset{w}{\min }(\Vert y-x\cdot w{\Vert }_2^2+\lambda \Vert w{\Vert }_1).$$

• When $\lambda =0$ , no parameters are eliminated. The estimate is equal to the one found with linear regression.
• As $\lambda$ increases, more and more coefficients are set to zero and eliminated (theoretically, when $\lambda =\infty$, all coefficients are eliminated).
• As $\lambda$ increases, bias increases.
• As $\lambda$ decreases, variance increases.

Lasso Regression : L2 Regularization

$$\underset{w}{\min }(\Vert y-x\cdot w{\Vert }_2^2+\lambda \Vert w{\Vert }_2^2).$$

$$L(w)=w^T{x}^{T}xw-w^T{x}^{T}y-y^{T}xw+y^{T}y+\lambda w^{T}w$$

$$\frac{\partial L(w)}{\partial w}=2x^{T}xw-2x^{T}y+2\lambda w$$

Let $\partial L(w)/\partial w=0$ leads to

$$w=(x^{T}x+\lambda I{)}^{-1}{x}^{T}y$$

Elastic net Regularization

$$\underset{w}{\min }(\Vert y-x\cdot w{\Vert }_2^2+{\lambda }_2\Vert w{\Vert }_2^2+{\lambda }_1\Vert w{\Vert }_1).$$

Logistic Regression

For $p\in (0,1)$,

$$\mathrm{logit}p=\ln (\mathrm{odds})=\ln \frac{p}{1-p}$$

Logistic function is inverse logit, sigmoid function defined as follows, for $q\in (-\infty ,\infty )$,

$$p=s(q)={\mathrm{logit}}^{-1}q=\frac{e^q}{e^q+1}=\frac{1}{1+e^{-q}}$$

Suppose $\mathrm{logit}y$ linear with $x$, i.e.

$$\ln \frac{y}{1-y}={\beta }_0+{\beta }_{1}x$$

then

$$y=s(x)=\frac{1}{1+e^{-({\beta }_0+{\beta }_{1}x)}}$$