Introduction to Deep Learning - Deep Neural Network
The progress of the deep learning field over the last couple of years is astounding. From creating vague images in 2015 to generating realistic videos with just a few minutes of prompts, the changes are truly remarkable.
- Introduction to Deep Learning - Deep Neural Network ← you are here
- Introduction to Deep Learning - Deep Sequence Model
AI vs ML vs deep learning
Deep learning aims to extract features from the data. Engineering the features manually is time-consuming, error-prone, and unscalable. Hence, neural networks are used to extract different levels of features in a lower-dimensional scenario.
Perceptron
Perceptron
A single unit of a neuron in a neural network is referred to as a perceptron. The purpose of the activation function is to introduce non-linearity in the network. This non-linearity enables the network to learn complex models with ease, which would otherwise require a larger network.
Perceptron: deep neural nw
Loss & Gradients
The loss of our network measures the cost of incorrect predictions. Empirical loss measures the total loss across the dataset.
Mean-squared error (continuous targets):
\[\mathcal{L}_{\mathrm{MSE}} = \frac{1}{N}\sum_{i=1}^{N}\bigl(y_i - \hat{y}_i\bigr)^2\]Binary cross-entropy (models that output a probability $\hat{y}\in(0,1)$):
\[\mathcal{L}_{\mathrm{CE}} = -\frac{1}{N}\sum_{i=1}^{N}\Bigl[y_i\log\hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\Bigr]\]We aim to determine network weights that yield the lowest loss. Gradient descent converges to a local minimum iteratively by stepping against the gradient:
\[w \leftarrow w - \eta\,\nabla_w\mathcal{L}(w)\]where $\eta$ is the learning rate and $\nabla_w\mathcal{L}$ is the gradient of the loss w.r.t. the weights.
Loss & Gradients: gradient descent
The way the gradients are calculated in a step-by-step manner from output to input direction is called backpropagation.
The gradient descent algorithm is inherently a greedy approach.
Use an adaptive learning rate $\eta$ to overcome the overshooting and undershooting problems commonly associated with fixed learning rates. Different gradient descent algorithms are proposed based upon different adaptive mechanisms. SGD and Adam are the most widely used variations of gradient descent.
Gradient calculation is expensive. Hence, it is generally computed over a batch.
Additionally, if the training data is small, there’s a possibility of model overfitting, where the network memorizes the training data rather than understanding the underlying features, which can increase the validation loss, even though the training loss is minimal. This can be discouraged in different ways. Regularization is one such mechanism; it simply nudges the network to use a simple model.
- During training, randomly set some activations to 0 with some probability. It prevents the network from depending on any specific node.
- Early stopping: Stop training before we can have a chance of overfitting
Loss & Gradients: early stopping
Concepts covered
Concepts covered: overview