Chapter 9: Single-Head Attention - Tokens Looking at Each Other

What You'll Build The attention mechanism: the only place in a transformer where a token at position t gets to look at tokens at positions 0..t-1 . This is specifically self-attention , where the token attends to other tokens in the same sequence. (You might encounter "cross-attention" in other materials, which is used in encoder-decoder models where tokens attend to a different sequence. We don't use cross-attention here.) Depends On Chapters 1-2, 5 (Value, Helpers). The Core Idea Until now, ea