Understanding Decoder-Only Transformers Part 1: Masked Self-Attention

Decoder-Only Transformers In this article, we will explore decoder-only transformers . Decoder-only transformers are a specific type of transformer architecture used in systems like ChatGPT. Masked Self-Attention Decoder-only transformers use a mechanism called masked self-attention . Masked self-attention works by measuring how similar each word is to itself and to the words that come before it in the sentence. For example: “The pizza came out of the oven and it tasted good.” When processing th