Deep Learning (Fach) / Recurrent Neural Networks (RNNs) (Lektion)

Diese Lektion wurde von Taggotty erstellt.

Lektion lernen

zurück  |  weiter 1 / 1

• What kind of recurrent connections are there in RNNs? - Direct feedback     There exists a connection from a neuron j to itself with weight wj,j - Indirect feedback     Activity is sent back to the previous layer in the neural network - Lateral feedback     Connect neurons within layers
• What is a bidirectional RNN? It has 2 hidden layers: - scan the input sequences separately - one forward and one backword - connect the same output layer - learn both past and future context
• How is feedbak included into the network structure? The output of the hidden layer is stored in the memory, and considered as additional input in the next time step. ht - current state fw - function with parameters W ht-1 - state in memory ( previous state ) xt - current input vector ht = fw(ht-1, xt) ht = tanh(Whhht-1 + Wxhxt) yt = Whyht
• What is an Unrolled RNN? - Reuse the same weight matrix at every time step. -> Makes the network easier to train
• How are the weights updated in RNNs? - Backpropagation through tim (BPTT) - Forward: Run through the entire sequence to compute the loss - Backward: Run through the entire sequence to compute the gradient and update the weight matrix w <- w - ηδL/δw Every path from W to Loss is one dependency. All paths from W to Loss need to be involved. We need to sum up the gradients at each time step δL/δw=sum(j=0, T, sum(k=1, j, div(δLj, δyj) * div(δyj, δh)j, (mul(t=k+1, j, div(δht, δht-1))) * div(δhk, δw)
• What are LSTMs? - Long Short-term Memory - the internal structure of an LSTM unit allows it to keep or forget information over time - Cell State: the memory of the network carries relevant information throughout the processing of the sequence - Information gets added or removed to the cell state via gates      - Forget gate: what is relevant to keep from prior step     - Input gate: what information is relevant to add from current step     - Output gate: what the next hidden state should be - The gates learn to determine what is reelvant during training 1. concat input and previous hidden units 2a. weighted sum with sigmoid activation (Forget gate) 2b. multiply with previous cell state 3aa. weighted sum of 1. with sigmoid activation (Input gate) 3ab. weighted sum of 1. with tanh activation (Input gate) 3b. multiply 3aa and 3ab 4. add 2b. to 3b. (next cell state) 5a. weighted sum of 1. with sigmoid activation (Output gate) 5b. tanh activation of 4. 5c. multiply 5a. and 5b. (next hidden units) 4a.
• What are properties of Vanilla RNNs? - Suffer from short term memory - If a sequence is long enough they cannot carry information from earlier steps to later - Vanishing/Exploding Gradients Concat Hidden Units with Inputs -> weighted sum -> tanh activation
• What is a GRU? - Gated Recurrent Unit - a simplified variant of the LSTM unit - GRUs only have two gates      - Update gate: Decides how much of previous memory to keep     - Reset gate: How to combine new input with previous value - The GRU has no persistent cell state Steps: 1. concatenate input and previous hidden units 2a. weighted sum of 1. with sigmoid activation (Reset gate) 2b. multiply hidden units with 2a. 3. weighted sum of 1. with sigmoid activation (Update gate) 4a. convatenate input and 2b. 4b. weighted sum of 4a with tanh activation (Output gate) 5a. multiply 4b. with 3 (Update and Output) 5b. subtract 3 [0, 1] from 1 5c. multiply 5b with the previous hidden units 5d. add 5c and 5a (new hidden units)

zurück  |  weiter 1 / 1