sparse coding VS autoencoder_综合

Finding the differences can be done by looking at the models. Let's look at sparse coding first.

Sparse coding

Sparse coding minimizes the objective

L sc = | | W H ? X | | 22 ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? reconstruction term + λ | | H | | 1 ? ? ?? ? ? ? ? sparsity term

where W W is a matrix of bases, H is a matrix of codes and X X is a matrix of the data we wish to represent. λ λ implements a trade of between sparsity and reconstruction. Note that if we are given H H , estimation of W W is easy via least squares.

In the beginning, we do not have H however. Yet, many algorithms exist that can solve the objective above with respect to H . Actually, this is how we do inference: we need to solve an optimisation problem if we want to know the h belonging to an unseen x .

Auto encoders

Auto encoders are a family of unsupervised neural networks. There are quite a lot of them, e.g. deep auto encoders or those having different regularisation tricks attached--e.g. denoising, contractive, sparse. There even exist probabilistic ones, such as generative stochastic networks or the variational auto encoder. Their most abstract form is

D (d (e (x; θ r); θ d), x)

but we will go along with a much simpler one for now:

L ae = | | W σ (W T X) ? X | | 2

where σ σ is a nonlinear function such as the logistic sigmoid σ(x)=11+exp(?x) σ ( x ) = 1 1 + exp ? ( ? x ) .

Similarities

Note that Lsc looks almost like Lae once we set H=σ(WTX) . The difference of both is that i) auto encoders do not encourage sparsity in their general form ii) an autoencoder uses a model for finding the codes, while sparse coding does so by means of optimisation.

For natural image data, regularized auto encoders and sparse coding tend to yield very similar W . However, auto encoders are much more efficient and are easily generalized to much more complicated models. E.g. the decoder can be highly nonlinear, e.g. a deep neural network. Furthermore, one is not tied to the squared loss (on which the estimation of W for Lsc depends.)

Also, the different methods of regularisation yield representations with different characteristica. Denoising auto encoders have also been shown to be equivalent to a certain form of RBMs etc.

But why?

If you want to solve a prediction problem, you will not need auto encoders unless you have only little labeled data and a lot of unlabeled data. Then you will generally be better of to train a deep auto encoder and put a linear SVM on top instead of training a deep neural net.

However, they are very powerful models for capturing characteristica of distributions. This is vague, but research turning this into hard statistical facts is currently conducted. Deep latent Gaussian models aka Variational Auto encoders or generative stochastic networks are pretty interesting ways of obtaining auto encoders which provably estimate the underlying data distribution.