Deep Learning Fundamentals · Part 1 of 2

Introduction to Deep Learning - Deep Neural Network

Tisan Das · · 3 min read

Contents

    The progress of the deep learning field over the last couple of years is astounding. From creating vague images in 2015 to generating realistic videos with just a few minutes of prompts, the changes are truly remarkable.

    AI vs ML vs deep learning AI vs ML vs deep learning AI vs ML vs deep learning

    Deep learning aims to extract features from the data. Engineering the features manually is time-consuming, error-prone, and unscalable. Hence, neural networks are used to extract different levels of features in a lower-dimensional scenario.

    Perceptron

    Perceptron Perceptron Perceptron

    A single unit of a neuron in a neural network is referred to as a perceptron. The purpose of the activation function is to introduce non-linearity in the network. This non-linearity enables the network to learn complex models with ease, which would otherwise require a larger network.

    Perceptron: deep neural nw Perceptron: deep neural nw Perceptron: deep neural nw

    Loss & Gradients

    The loss of our network measures the cost of incorrect predictions. Empirical loss measures the total loss across the dataset.

    Mean-squared error (continuous targets):

    LMSE=1N∑i=1N(yi−y^i)2\mathcal{L}_{\mathrm{MSE}} = \frac{1}{N}\sum_{i=1}^{N}\bigl(y_i - \hat{y}_i\bigr)^2

    Binary cross-entropy (models that output a probability y^∈(0,1)\hat{y}\in(0,1)):

    LCE=−1N∑i=1N[yilog⁡y^i+(1−yi)log⁡(1−y^i)]\mathcal{L}_{\mathrm{CE}} = -\frac{1}{N}\sum_{i=1}^{N}\Bigl[y_i\log\hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\Bigr]

    We aim to determine network weights that yield the lowest loss. Gradient descent converges to a local minimum iteratively by stepping against the gradient:

    w←w−η ∇wL(w)w \leftarrow w - \eta\,\nabla_w\mathcal{L}(w)

    where η\eta is the learning rate and ∇wL\nabla_w\mathcal{L} is the gradient of the loss w.r.t. the weights.

    Loss & Gradients: gradient descent Loss & Gradients: gradient descent Loss & Gradients: gradient descent

    The way the gradients are calculated in a step-by-step manner from output to input direction is called backpropagation.

    The gradient descent algorithm is inherently a greedy approach.

    Use an adaptive learning rate η\eta to overcome the overshooting and undershooting problems commonly associated with fixed learning rates. Different gradient descent algorithms are proposed based upon different adaptive mechanisms. SGD and Adam are the most widely used variations of gradient descent.

    Gradient calculation is expensive. Hence, it is generally computed over a batch.

    Additionally, if the training data is small, there’s a possibility of model overfitting, where the network memorizes the training data rather than understanding the underlying features, which can increase the validation loss, even though the training loss is minimal. This can be discouraged in different ways. Regularization is one such mechanism; it simply nudges the network to use a simple model.

    1. During training, randomly set some activations to 0 with some probability. It prevents the network from depending on any specific node.
    2. Early stopping: Stop training before we can have a chance of overfitting

    Loss & Gradients: early stopping Loss & Gradients: early stopping Loss & Gradients: early stopping

    Concepts covered

    Concepts covered: overview Concepts covered: overview Concepts covered: overview

    References:

    1. MIT Introduction to Deep Learning 6.S191

    Open mail app

    Write in Gmail · Outlook