Autoencoder’s are Deeplearning’s approach to dimensionality reduction ,if you are unfamiliar about what is dimensionality reduction click here . Autoencoder’s research paper was published somewhere around 2009-2010, and from that it is proven one of the best techniques for dimensionality reduction.Autoencoder’s works almost better than TSNE (Another dimensionality reduction algorithm which uses t-distribution to preserve neighborhood of a point).
The goal of this article is to answer these questions,
- What is Autoencoder ?
- How does it works ?
- How to implement it using python?
Before proceeding further make sure you understand the working of a neural network, if not then click here
What is Autoencoder ?
An Autoencoder is a neural network architecture used for dimensionality reduction ,the idea behind autoencoder is very simple first try to reduce the dimension then from the reduce dimension try to recreate the original dimension.This approach can easily be understood using the Figure below.
In this the Z is the resultant dimension in which we want to convert the original dimension x and x’ is recreated dimension from Z.
How does it woks?
For demonstration of this we will use one of the simplest Autoencoder architecture ,given below
Let’s understand using it’s algorithm.
- input X
- Create hidden layer of size equal to reduced dimension(i.e your choice of dimension) with linear activation function i.e f(z) = Z
- Then create a output layer equal to the original dimension.
- Pass the created x and original x in loss function in our case is simple square of L-2 norm ||||
Note: the network is fully connected with no dropout as eliminating any neuron makes no sense.
Training the network
- First multiply(i.e dot product) the weight matrix of hidden layer to the input Z= W.t*X +b.
- Pass it through activation function in our case it is a linear activation function i.e it outputs Z only.
- Then multiply (i.e) the Z with Output layer matrix which is of size same as that of input .
- Compute Loss
- BackPropagate the loss inside the network to update the weights (don’t know backpropagation click here).
Autoencoder’s are very powerful tool and can do wonder’s which were not possible before take example of a popular neural network architecture called Deepfakes which can be used to change faces from a video character with any other .It uses a combination of CNN and autoencoders to do this.
We will cover many such specific neural network architecture so stay connected.