Attention and Transformers from Scratch

Get Started

The voice-over in this course is synthesized with a text-to-speech model from scripts written and reviewed by the instructor, and the on-screen material (notebooks, code, slides) is the instructor’s own work. Attention is the idea behind every modern language model, and it was invented to solve one concrete problem: a sequence-to-sequence model squeezes a whole sentence into one fixed vector, and long sentences fall apart. This course starts from that problem and builds all the way up to a working Transformer.

We begin with what language models can do and the roadmap of an NLP engineer. Then we implement an RNN Seq2Seq model in PyTorch and measure its limits: the fixed context vector, what a wider hidden state does and does not fix. We derive the attention mechanism in depth — the core idea, the alignment weights, Bahdanau’s additive attention — and implement Bahdanau and Luong attention with alignment visualisations. Next we read “Attention Is All You Need” section by section: self-attention, multi-head attention, positional encoding, feed-forward blocks, encoder and decoder. Finally we assemble a Transformer from parts — positional encoding, masking, multi-head attention, Add & Norm, encoder and decoder layers — train it, translate with it, and inspect cross-attention.