Lasso算法

Some basic properties of the lasso estimator can now be considered.

Assuming first that the covariates are orthonormal so that ${\displaystyle (x_{i}\mid x_{j})=\delta _{ij}}$, where ${\displaystyle (\cdot \mid \cdot )}$ is the inner product and ${\displaystyle \delta _{ij}}$ is the Kronecker delta, or, equivalently, ${\displaystyle X^{T}X=I}$, then using subgradient methods it can be shown that

${\displaystyle {\begin{aligned}{\hat {\beta }}_{j}={}&S_{N\lambda }({\hat {\beta }}_{j}^{\text{OLS}})={\hat {\beta }}_{j}^{\text{OLS}}\max \left(0,1-{\frac {N\lambda }{|{\hat {\beta }}_{j}^{\text{OLS}}|}}\right)\\&{\text{ where }}{\hat {\beta }}^{\text{OLS}}=(X^{T}X)^{-1}X^{T}y\end{aligned}}}$ [1]

${\displaystyle S_{\alpha }}$ is referred to as the soft thresholding operator, since it translates values towards zero (making them exactly zero if they are small enough) instead of setting smaller values to zero and leaving larger ones untouched as the hard thresholding operator, often denoted ${\displaystyle H_{\alpha }}$, would.

This can be compared to ridge regression, where the objective is to minimize

${\displaystyle \min _{\beta \in \mathbb {R} ^{p}}\left\{{\frac {1}{N}}\|y-X\beta \|_{2}^{2}+\lambda \|\beta \|_{2}^{2}\right\}}$

yielding

${\displaystyle {\hat {\beta }}_{j}=(1+N\lambda )^{-1}{\hat {\beta }}_{j}^{\text{OLS}}.}$

So ridge regression shrinks all coefficients by a uniform factor of ${\displaystyle (1+N\lambda )^{-1}}$ and does not set any coefficients to zero.

It can also be compared to regression with best subset selection, in which the goal is to minimize

${\displaystyle \min _{\beta \in \mathbb {R} ^{p}}\left\{{\frac {1}{N}}\left\|y-X\beta \right\|_{2}^{2}+\lambda \|\beta \|_{0}\right\}}$

where ${\displaystyle \|\cdot \|_{0}}$ is the "${\displaystyle \ell ^{0}}$ norm", which is defined as ${\displaystyle \|z\|=m}$ if exactly m components of z are nonzero. In this case, it can be shown that

${\displaystyle {\hat {\beta }}_{j}=H_{\sqrt {N\lambda }}\left({\hat {\beta }}_{j}^{\text{OLS}}\right)={\hat {\beta }}_{j}^{\text{OLS}}\mathrm {I} \left(\left|{\hat {\beta }}_{j}^{\text{OLS}}\right|\geq {\sqrt {N\lambda }}\right)}$

where ${\displaystyle H_{\alpha }}$ is the so-called hard thresholding function and ${\displaystyle \mathrm {I} }$ is an indicator function (it is 1 if its argument is true and 0 otherwise).

Therefore, the lasso estimates share features of the estimates from both ridge and best subset selection regression since they both shrink the magnitude of all the coefficients, like ridge regression, but also set some of them to zero, as in the best subset selection case. Additionally, while ridge regression scales all of the coefficients by a constant factor, lasso instead translates the coefficients towards zero by a constant value and sets them to zero if they reach it.

Returning to the general case, in which the different covariates may not be independent, a special case may be considered in which two of the covariates, say j and k, are identical for each case, so that ${\displaystyle x_{(j)}=x_{(k)}}$, where ${\displaystyle x_{(j),i}=x_{ij}}$. Then the values of ${\displaystyle \beta _{j}}$ and ${\displaystyle \beta _{k}}$ that minimize the lasso objective function are not uniquely determined. In fact, if there is some solution ${\displaystyle {\hat {\beta }}}$ in which ${\displaystyle {\hat {\beta }}_{j}{\hat {\beta }}_{k}\geq 0}$, then if ${\displaystyle s\in [0,1]}$ replacing ${\displaystyle {\hat {\beta }}_{j}}$ by ${\displaystyle s({\hat {\beta }}_{j}+{\hat {\beta }}_{k})}$ and ${\displaystyle {\hat {\beta }}_{k}}$ by ${\displaystyle (1-s)({\hat {\beta }}_{j}+{\hat {\beta }}_{k})}$, while keeping all the other ${\displaystyle {\hat {\beta }}_{i}}$ fixed, gives a new solution, so the lasso objective function then has a continuum of valid minimizers.[5] Several variants of the lasso, including the Elastic Net, have been designed to address this shortcoming, which are discussed below.