Deep Learning From Scratch - Theory and Implementation

DanielSabinasz

360.6K views

Open Source Your Knowledge, Become a Contributor

Technology knowledge has to be shared and made accessible for free. Join the movement.

Create Content

Previous: Gradient Descent and Backpropagation Next: TensorFlow

Multi-layer perceptrons

Motivation

So now we are able to train linear classifiers of arbitrary dimensionality automatically. However, many real-world classes are not linearly separable. This means that there does not exist any line with all the points of the first class on one side of the line and all the points of the other class on the other side. Let's illustrate this with an example.

Example

# Create two clusters of red points centered at (0, 0) and (1, 1), respectively.
red_points = np.concatenate((
    0.2*np.random.randn(25, 2) + np.array([[0, 0]]*25),
    0.2*np.random.randn(25, 2) + np.array([[1, 1]]*25)
))
# Create two clusters of blue points centered at (0, 1) and (1, 0), respectively.
blue_points = np.concatenate((
    0.2*np.random.randn(25, 2) + np.array([[0, 1]]*25),
    0.2*np.random.randn(25, 2) + np.array([[1, 0]]*25)
))

Example

As we can see, it is impossible to draw a line that separates the blue points from the red points. Instead, our decision boundary has to have a rather complex shape. This is where multi-layer perceptrons come into play: They allow us to train a decision boundary of a more complex shape than a straight line.

Computational graph

As their name suggests, multi-layer perceptrons (MLPs) are composed of multiple perceptrons stacked one after the other in a layer-wise fashion. Let's look at a visualization of the computational graph:

As we can see, the input is fed into the first layer, which is a multidimensional perceptron with a weight matrix $W_{1}$ and bias vector $b_{1}$ . The output of that layer is then fed into second layer, which is again a perceptron with another weight matrix $W_{2}$ and bias vector $b_{2}$ . This process continues for every of the $L$ layers until we reach the output layer. We refer to the last layer as the output layer and to every other layer as a hidden layer.

an MLP with one hidden layers computes the function

σ (σ (X W_{1} + b_{1}) W_{2} + b_{2}),

an MLP with two hidden layers computes the function

σ (σ (σ (X W_{1} + b_{1}) W_{2} + b_{2}) W_{3},

and, generally, an MLP with $L - 1$ hidden layers computes the function

σ (σ (\dots σ (σ (X W_{1} + b_{1}) W_{2} + b_{2}) \dots) W_{L} + b_{L}) .

Implementation

Using the library we have built, we can now easily implement multi-layer perceptrons without further work.

Multi-Layer Perceptron

# Create a new graph
Graph().as_default()
# Create training input placeholder
X = placeholder()
# Create placeholder for the training classes
c = placeholder()
# Build a hidden layer
W_hidden = Variable(np.random.randn(2, 2))
b_hidden = Variable(np.random.randn(2))
p_hidden = sigmoid(add(matmul(X, W_hidden), b_hidden))
# Build the output layer
W_output = Variable(np.random.randn(2, 2))
b_output = Variable(np.random.randn(2))
p_output = softmax(add(matmul(p_hidden, W_output), b_output))
# Build cross-entropy loss
J = negative(reduce_sum(reduce_sum(multiply(c, log(p_output)), axis=1)))
# Build minimization op
minimization_op = GradientDescentOptimizer(learning_rate=0.03).minimize(J)
# Build placeholder inputs
feed_dict = {
    X: np.concatenate((blue_points, red_points)),
    c:
        [[1, 0]] * len(blue_points)
        + [[0, 1]] * len(red_points)
}
# Create session
session = Session()
# Perform 100 gradient descent steps
for step in range(1000):
    J_value = session.run(J, feed_dict)
    if step % 100 == 0:
        print("Step:", step, " Loss:", J_value)
    session.run(minimization_op, feed_dict)
# Print final result
W_hidden_value = session.run(W_hidden)
print("Hidden layer weight matrix:\n", W_hidden_value)
b_hidden_value = session.run(b_hidden)
print("Hidden layer bias:\n", b_hidden_value)
W_output_value = session.run(W_output)
print("Output layer weight matrix:\n", W_output_value)

As we can see, we have learned a rather complex decision boundary. If we use more layers, the decision boundary can become arbitrarily complex, allowing us to learn classification patterns that are impossible to spot by a human being, especially in higher dimensions.

Recap

Congratulations on making it this far! You have learned the foundations of building neural networks from scratch, and in contrast to most machine learning practitioners, you now know how it all works under the hood and why it is done the way it is done.

Let's recap what we have learned. We started out by considering computational graphs in general, and we saw how to build them and how to compute their output. We then moved on to describe perceptrons, which are linear classifiers that assign a probability to each output class by squashing the output of $w^{T} x + b$ through a sigmoid (or softmax, in the case of multiple classes). Following that, we saw how to judge how good a classifier is - via a loss function, the cross-entropy loss, the minimization of which is equivalent to maximum likelihood. In the next step, we saw how to minimize the loss via gradient descent: By iteratively stepping into the direction of the negative gradient. We then introduced backpropagation as a means of computing the derivative of the loss with respect to each node by performing a breadth-first search and multiplying according to the chain rule. We used all that we've learned to train a good linear classifier for the red/blue example dataset. Finally, we learned about multi-layer perceptrons as a means of learning non-linear decision boundaries, implemented an MLP with one hidden layer and successfully trained it on a non-linearly-separable dataset.

Next steps

You now know all the fundamentals for training arbitrary neural networks. As a next step, you should learn about the following topics (Google is your friend):

The difference between training loss and test loss
Overfitting and underfitting
Regularization and early stopping
Dropout
Convolutional neural networks
Recurrent neural networks
Autoencoders
Deep Generative Models

All of these topics are dealt with in the book "Deep Learning" by Ian Goodfellow, Yoshua Bengio and Aaron Courville, which I highly recommend everyone to read. A free online version of the book can be found at http://www.deeplearningbook.org/.

Open Source Your Knowledge: become a Contributor and help others learn. Create New Content

Open Source Your Knowledge, Become a Contributor

5/6 Multi-Layer Perceptrons

Multi-layer perceptrons

Motivation

Computational graph

Implementation

Recap

Next steps

Handwritten Digit Recognition Using scikit-learn

PYTHON: BEGINNER QUIZ (10 Questions)

Simple Python Test

PYTHON STRING METHODS: center(), rjust() and ljust().