Understanding Attention Mechanisms – Part 3: From...

In the previous article, we explored the comparison between encoder and decoder outputs. In this article, we will be checking the math on how the calculation is done, and how it can be further simplified. The output values for the two LSTM cells in the encoder for the word "Let’s" are -0.76 and 0.75. The output values from the two LSTM cells in the decoder for the <EOS> token are 0.91 and 0.38. We can represent this as: A = Encoder B = Decoder Cell #1 Cell #2 -0.76 0.75 0.91 0.38 Now, we plug th