Introduction to Deep Learning - Deep Sequence Model
Given an image of a ball, how do we predict where it will go next? The traditional models we’ve seen previously in the previous part don’t have any notion of sequence. Hence, even if we try to use the feedforward network models we’ve seen previously on a timeseries data, they would treat the data points as a slice of data in a particular timestamp, and wouldn’t be able to capture the sequential nature.
- Introduction to Deep Learning - Deep Neural Network
- Introduction to Deep Learning - Deep Sequence Model ← you are here
The sequential feature can be captured by introducing another set of hidden states to maintain prior history, as we move through the sequence. The primary objective of these hidden states is to serve as a previous memory, updating with each timestamp input. These are also called recurrence relations, and the network is called Recurrent Neural Network.
The hidden state $h_t$ and output $\hat{y}_t$ at time $t$ are:
\[h_t = f\bigl(W_{hh}h_{t-1} + W_{xh}x_t + b_h\bigr), \qquad \hat{y}_t = g\bigl(W_{hy}h_t + b_y\bigr)\]where $f$ and $g$ are activation functions (e.g. $\tanh$, softmax), and the $W_{\cdot}$ matrices are shared across all timesteps.
Recurrence network 
Recurrence network calc
Compute the loss of the network at the individual slice, and then the total loss by summing over all the timestamps in our sequence of length $T$:
\[\mathcal{L} = \sum_{t=1}^{T}\mathcal{L}_t\bigl(y_t, \hat{y}_t\bigr)\]
Recurrence network loss
Design criteria of sequence models:
- Handle variable-length sequence
- Track long-term dependencies (something at the very beginning dictates something at the very end)
- Maintain information about the order
- Share parameters across sequences
Real-life example: Predict the next word
Encoding language for a Neural Network:
- Create a vocabulary of all possible words/sub-words
- Map each word into an index, so that the Neural network can be trained on a series of numerical values
- Embedding: transform indexes into a vector of fixed size - Neural networks are trained to embed indexes in a fixed-dimensional space, so that similar words are placed together
In order to handle the time-dependence, the backpropagation algorithm also needs to aggregate the loss over the individual time slices. This is also known as Backpropagation Through Time.
Encoding language for a Neural Network: recurrence network backpropagation
The issue with this type of backpropagation algorithm is that the gradient multiplies through many timesteps, so the product of Jacobians can grow or shrink exponentially:
- Exploding gradients: many factors $\lvert\partial h_t/\partial h_{t-1}\rvert > 1$
- Vanishing gradients: many factors $\lvert\partial h_t/\partial h_{t-1}\rvert < 1$
Encoding language for a Neural Network: recurrence network vaishing gradients
One way to handle the issue of exploding and vanishing gradients is to use gates to selectively add or remove information within each recurrent unit.
Limitations of RNN:
- Slow, no parallelism
- Not long memory