Deep Learning Fundamentals · Part 2 of 2

Introduction to Deep Learning - Deep Sequence Model

Tisan Das · · 3 min read

Contents

    Given an image of a ball, how do we predict where it will go next? The traditional models we’ve seen previously in the previous part don’t have any notion of sequence. Hence, even if we try to use the feedforward network models we’ve seen previously on a timeseries data, they would treat the data points as a slice of data in a particular timestamp, and wouldn’t be able to capture the sequential nature.

    The sequential feature can be captured by introducing another set of hidden states to maintain prior history, as we move through the sequence. The primary objective of these hidden states is to serve as a previous memory, updating with each timestamp input. These are also called recurrence relations, and the network is called Recurrent Neural Network.

    The hidden state hth_t and output y^t\hat{y}_t at time tt are:

    ht=f(Whhht−1+Wxhxt+bh),y^t=g(Whyht+by)h_t = f\bigl(W_{hh}h_{t-1} + W_{xh}x_t + b_h\bigr), \qquad \hat{y}_t = g\bigl(W_{hy}h_t + b_y\bigr)

    where ff and gg are activation functions (e.g. tanh⁡\tanh, softmax), and the W⋅W_{\cdot} matrices are shared across all timesteps.

    Recurrence network Recurrence network Recurrence network Recurrence network calc Recurrence network calc Recurrence network calc

    Compute the loss of the network at the individual slice, and then the total loss by summing over all the timestamps in our sequence of length TT:

    L=∑t=1TLt(yt,y^t)\mathcal{L} = \sum_{t=1}^{T}\mathcal{L}_t\bigl(y_t, \hat{y}_t\bigr)

    Recurrence network loss Recurrence network loss Recurrence network loss

    Design criteria of sequence models:

    • Handle variable-length sequence
    • Track long-term dependencies (something at the very beginning dictates something at the very end)
    • Maintain information about the order
    • Share parameters across sequences

    Real-life example: Predict the next word

    Encoding language for a Neural Network:

    • Create a vocabulary of all possible words/sub-words
    • Map each word into an index, so that the Neural network can be trained on a series of numerical values
    • Embedding: transform indexes into a vector of fixed size    - Neural networks are trained to embed indexes in a fixed-dimensional space, so that similar words are placed together

    In order to handle the time-dependence, the backpropagation algorithm also needs to aggregate the loss over the individual time slices. This is also known as Backpropagation Through Time.

    Encoding language for a Neural Network: recurrence network backpropagation Encoding language for a Neural Network: recurrence network backpropagation Encoding language for a Neural Network: recurrence network backpropagation

    The issue with this type of backpropagation algorithm is that the gradient multiplies through many timesteps, so the product of Jacobians can grow or shrink exponentially:

    • Exploding gradients: many factors ∣∂ht/∂ht−1∣>1\lvert\partial h_t/\partial h_{t-1}\rvert > 1
    • Vanishing gradients: many factors ∣∂ht/∂ht−1∣<1\lvert\partial h_t/\partial h_{t-1}\rvert < 1

    Encoding language for a Neural Network: recurrence network vaishing gradients Encoding language for a Neural Network: recurrence network vaishing gradients Encoding language for a Neural Network: recurrence network vaishing gradients

    One way to handle the issue of exploding and vanishing gradients is to use gates to selectively add or remove information within each recurrent unit.

    Limitations of RNN:

    • Slow, no parallelism
    • Not long memory

    References:

    1. MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention

    Open mail app

    Write in Gmail · Outlook