Recurrent neural networks Modelling sequential data. MLP Lecture 9 Recurrent Networks 1

Size: px

Start display at page:

Download "Recurrent neural networks Modelling sequential data. MLP Lecture 9 Recurrent Networks 1"

Lynette Gray
6 years ago
Views:

1 Recurrent neural networks Modelling sequential data MLP Lecture 9 Recurrent Networks 1

2 Recurrent Networks Steve Renals Machine Learning Practical MLP Lecture 9 16 November 2016 MLP Lecture 9 Recurrent Networks 2

3 Introduction - Recurrent Neural Networks (RNNs) Modelling sequential data Recurrent hidden unit connections Training RNNs: Back-propagation through time LSTMs Examples (speech and language) MLP Lecture 9 Recurrent Networks 3

4 Sequential Data output Modelling sequential data with time dependences between feature vectors hidden input x1 x2 x3 x1 x2 x3 x1 x2 x3 t-2 t-1 t 2 frames of context MLP Lecture 9 Recurrent Networks 4

5 Sequential Data input output hidden x1 x2 x3 x1 x2 x3 t-2 t-1 2 frames of context x1 x2 x3 t Modelling sequential data with time dependences between feature vectors Can model fixed context with a feed-forward network with previous time input vectors added to the network input Finite context determined by window width MLP Lecture 9 Recurrent Networks 4

6 Sequential Data recurrent hidden output input x1 x2 x3 t Modelling sequential data with time dependences between feature vectors Can model fixed context with a feed-forward network with previous time input vectors added to the network input Finite context determined by window width Model sequential inputs using recurrent connections to learn a time-dependent state Potentially infinite context MLP Lecture 9 Recurrent Networks 4

7 Recurrent networks If there was no external input... think of recurrent networks in terms of the dynamics of the recurrent hidden state Settle to a fixed point stable representation Regular oscillation ( limit cycle ) learn some kind of repetition Chaotic dynamics (non-repetitive) theoretically interesting ( computation at the edge of chaos ) Useful behaviours of recurrent networks with external inputs: Recurrent state as memory remember things for (potentially) an infinite time Recurrent state as information compression compress a sequence into a state representation MLP Lecture 9 Recurrent Networks 5

8 Vanilla RNNs MLP Lecture 9 Recurrent Networks 6

9 Simplest recurrent network y k (t) = softmax ( H r=0 d h j (t) = sigmoid s=0 w (2) kr h r (t) b k w (1) js x s (t) H r=0 ) jr h r (t 1) b j w (R) } {{ } Recurrent part Output (t) w (2) Hidden (t) w (1) w (R) Input (t) Hidden (t-1) MLP Lecture 9 Recurrent Networks 7

10 Recurrent network unfolded in time Output (t-1) Output (t) Output (t1) w (2) w (2) w (2) w (R) Hidden (t-1) w (R) Hidden (t) w (R) Hidden (t1) w (R) w (1) w (1) w (1) Input (t-1) Input (t) Input (t1) An RNN for a sequence of T inputs can be viewed as a deep T -layer network with shared weights MLP Lecture 9 Recurrent Networks 8

11 Recurrent network unfolded in time Output (t-1) Output (t) Output (t1) w (2) w (2) w (2) w (R) Hidden (t-1) w (R) Hidden (t) w (R) Hidden (t1) w (R) w (1) w (1) w (1) Input (t-1) Input (t) Input (t1) An RNN for a sequence of T inputs can be viewed as a deep T -layer network with shared weights MLP Lecture 9 Recurrent Networks 8

12 Recurrent network unfolded in time Output (t-1) Output (t) Output (t1) w (2) w (2) w (2) w (R) Hidden (t-1) w (R) Hidden (t) w (R) Hidden (t1) w (R) w (1) w (1) w (1) Input (t-1) Input (t) Input (t1) An RNN for a sequence of T inputs can be viewed as a deep T -layer network with shared weights We can train an RNN by doing backprop through this unfolded network, making sure we share the weights Weight sharing if two weights are constrained to be equal (w 1 = w 2 ) then they will stay equal if the weight changes are equal ( E/ w 1 = E/ w 2 ) achieve this by updating with ( E/ w 1 E/ w 2 ) (cf Conv Nets) MLP Lecture 9 Recurrent Networks 8

13 Back-propagation through time (BPTT) We can train a network by unfolding and back-propagating through time, summing the derivatives for each weight as we go through the sequence More efficiently, run as a recurrent network cache the unit outputs at each timestep cache the output errors at each timestep then backprop from the final timestep to zero, computing the derivatives at each step compute the weight updates by summing the derivatives across time Expensive backprop for a 1,000 item sequence equivalent to a 1,000-layer feed-forward network Truncated BPTT backprop through just a few time steps (e.g. 20) MLP Lecture 9 Recurrent Networks 9

14 Vanishing and exploding gradients BPTT involves taking the product of many gradients (as in a very deep network) this can lead to vanishing (component gradients less than 1) or exploding (greater than 1) gradients This can prevent effective training Modified optimisation algorithms RMSProp (and similar algorithms) normalise the gradient for each weight by average of it magnitude, with a learning rate for each weight Hessian-free an approximation to second-order approaches which use curvature information Modified hidden unit transfer functions Long short term memory (LSTM) Linear self-recurrence for each hidden unit (long-term memory) Gates - dynamic weights which are a function of their inputs Gated recurrent units MLP Lecture 9 Recurrent Networks 10

15 LSTM MLP Lecture 9 Recurrent Networks 11

16 Vanilla RNN h(t) g(t) Whh Whx x(t) g(t) = W hx x(t) W hh h(t 1) b h h(t) = tanh (g(t)) MLP Lecture 9 Recurrent Networks 12

17 LSTM Internal recurrent state ( cell ) c(t) combines previous state c(t 1) and LSTM input g(t) MLP Lecture 9 Recurrent Networks 13

18 LSTM h(t) g(t) W hh W hx x(t) MLP Lecture 9 Recurrent Networks 14

19 LSTM Internal recurrent state h(t) c(t-1) c(t) g(t) W hh W hx x(t) MLP Lecture 9 Recurrent Networks 14

20 LSTM Internal recurrent state ( cell ) c(t) combines previous state c(t 1) and LSTM input g(t) Gates - weights dependent on the current input and the previous state Input gate: controls how much input to the unit g(t) is written to the internal state c(t) Forget gate: controls how much of the previous internal state c(t 1) is written to the internal state c(t) Input and forget gates together allow the network to control what information is stored and overwritten at each step MLP Lecture 9 Recurrent Networks 15

21 LSTM h(t) c(t-1) c(t) g(t) W hh W hx x(t) MLP Lecture 9 Recurrent Networks 16

22 LSTM Input Gate h(t) c(t-1) c(t) I(t; x(t), ) g(t) W hh W hx x(t) MLP Lecture 9 Recurrent Networks 16

23 LSTM Forget Gate h(t) c(t-1) F(t; x(t), ) c(t) I(t; x(t), ) g(t) W hh W hx x(t) MLP Lecture 9 Recurrent Networks 16

24 LSTM Input and Forget Gates h(t) c(t-1) F(t; x(t), ) c(t) I(t; x(t), ) g(t) Whh Whx I(t) = σ (W ix x(t) W ih h(t 1) b i ) F(t) = σ (W fx x(t) W fh h(t 1) b f ) σ is the sigmoid function x(t) g(t) = W hx x(t) W hh h(t 1) b h c(t) = F(t) c(t 1) I(t) g(t) is element-wise vector multiply MLP Lecture 9 Recurrent Networks 17

25 LSTM Internal recurrent state ( cell ) c(t) combines previous state c(t 1) and LSTM input g(t) Gates - weights dependent on the current input and the previous state Input gate: controls how much input to the unit g(t) is written to the internal state c(t) Forget gate: controls how much of the previous internal state c(t 1) is written to the internal state c(t) Input and forget gates together allow the network to control what information is stored and overwritten at each step Output gate: controls how much of each unit s activation is output by the hidden state it allows the LSTM cell to kepp information that is not relevant at the current time, but may be relevant later MLP Lecture 9 Recurrent Networks 18

26 LSTM Input and Forget Gates h(t) c(t-1) F(t; x(t), ) c(t) I(t; x(t), ) g(t) W hh W hx x(t) MLP Lecture 9 Recurrent Networks 19

27 LSTM Output Gate h(t) c(t-1) F(t; x(t), ) O(t; x(t), ) c(t) I(t; x(t), ) g(t) W hh W hx x(t) MLP Lecture 9 Recurrent Networks 19

28 LSTM Output Gate h(t) c(t-1) F(t; x(t), ) O(t; x(t), ) c(t) I(t; x(t), ) g(t) Whh Whx O(t) = σ (W oxx(t) W oh h(t 1) b o) x(t) h(t) = tanh (O(t) c(t)) MLP Lecture 9 Recurrent Networks 20

29 LSTM h(t) c(t-1) F(t; x(t), ) O(t; x(t), ) c(t) I(t; x(t), ) g(t) Whh Whx x(t) I(t) = σ (W ix x(t) W ih h(t 1) b i ) F(t) = σ (W fx x(t) W fh ht 1) b f ) O(t) = σ (W oxx(t) W oh h(t 1) b o) g(t) = W hx x(t) W hh h(t 1) b h c(t) = F(t) c(t 1) I(t) g(t) h(t) = tanh (O(t) c(t)) MLP Lecture 9 Recurrent Networks 21

30 Example applications using RNNs MLP Lecture 9 Recurrent Networks 22

31 Example 1: speech recognition with recurrent networks Phoneme Probabilities Recurrent Neural Network freq (Hz) Speech Acoustics time (ms) T Robinson et al (1996). The use of recurrent networks in continuous speech recognition, in Automatic Speech and Speaker Recognition Advanced Topics (Lee et al (eds)), Kluwer, MLP Lecture 9 Recurrent Networks 23

32 Example 2: speech recognition with stacked LSTMs input input input input LSTM LSTM LSTM LSTM output (a) LSTM LSTM output recurrent output (c) LSTMP recurrent LSTM (b) DLSTM recurrent output (d) DLSTMP H Sak et al (2014). Long Short-Term Memory based Recurrent Neural Network Architectures for Large Scale Acoustic Modelling, Interspeech. MLP Lecture 9 Recurrent Networks 24

33 Example 3: recurrent network language models T Mikolov et al (2010). Recurrent Neural Network Based Language Model, Interspeech MLP Lecture 9 Recurrent Networks 25

34 Example 4: recurrent encoder-decoder Machine translation I Sutskever et al (2014). Sequence to Sequence Learning with Neural Networks, NIPS. K Cho et al (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, EMNLP. MLP Lecture 9 Recurrent Networks 26

35 Summary RNNs can model sequences Unfolding an RNN gives a deep feed-forward network Back-propagation through time LSTM More on recurrent networks next semester in NLU (and 1-2 lectures in ASR and MT) MLP Lecture 9 Recurrent Networks 27

Recurrent neural networks Modelling sequential data. MLP Lecture 9 / 13 November 2018 Recurrent Neural Networks 1: Modelling sequential data 1

Recurrent neural networks Modelling sequential data MLP Lecture 9 / 13 November 2018 Recurrent Neural Networks 1: Modelling sequential data 1 Recurrent Neural Networks 1: Modelling sequential data Steve