Public demo

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (2017)

Introduces the Transformer, a sequence model built entirely on attention—no recurrence or convolution—making training far more parallel while setting new translation records.

Why it matters

This is the architecture every modern large language model is built on. By showing that attention alone could model language, it removed the sequential bottleneck of RNNs, unlocked GPU-scale parallel training, and became the substrate for BERT, GPT, and essentially all of today's NLP.

Highlights

  • [method] The Transformer uses stacked self-attention and point-wise feed-forward layers in both encoder and decoder, with no recurrence or convolution. — why: this is the core architectural break that defines the paper.
  • [method] Multi-head attention lets the model attend to information from different representation subspaces at different positions in parallel. — why: the mechanism that makes attention expressive enough to replace recurrence.
  • [finding] On WMT 2014 English-to-German the model reached 28.4 BLEU, a new state of the art, while training in a fraction of the time of prior recurrent/convolutional models. — why: the empirical proof that "attention is all you need."
  • [limitation] Self-attention is O(n²) in sequence length, so very long sequences are expensive—a constraint that later long-context work (e.g. self-rag's era) keeps pushing on. — why: the honest cost behind the speed gains.

Reader notes

  • The real unlock isn't just accuracy—it's *parallelism*. Removing the step-by-step dependency of RNNs is what made training at today's scale feasible.
  • Positional encodings are how a non-recurrent model recovers word order; worth contrasting with the learned positions used later in bert.
  • Read this before anything else in the demo: bert, retrieval models, and agents all assume the Transformer block.

Extracted evidence examples

  • OBJECTIVE: The Transformer, a novel architecture solely using attention, abandons recurrence and convolutions for sequence transduction, demonstrating its importance and generalization capabilities.
    In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.
    To evaluate the importance of different components of the Transformer, we varied our base model in different ways, measuring the change in performance on English-to-German translation on the .
  • OBJECTIVE: The paper considers three desiderata motivating the use of self-attention layers.
    Motivating our use of self-attention we consider three desiderata.
  • OBJECTIVE: A research goal is to make the generation process less sequential, improving efficiency and potentially quality.
    Making generation less sequential is another research goals of ours.
  • RESEARCH_PROBLEM: Self-attention models, unlike recurrent or convolutional ones, excel at long-range dependencies in sequence transduction by enabling parallelization, crucial for complex tasks with long sequences.
    This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples.
    Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter.
  • RESEARCH_PROBLEM: Preventing leftward information flow in the decoder is necessary to maintain the auto-regressive property.
    We need to prevent leftward information flow in the decoder to preserve the auto-regressive property.
  • RESEARCH_PROBLEM: The application of law should be just, even though the law itself may not be perfect.
    The Law will never be perfect , but its application should be just this is what we are missing , in my opinion .
  • RESEARCH_GAP: Self-attention, unlike RNNs and CNNs, enables constant-time, all-pairs position connections, overcoming limitations in parallelization and long-range dependency learning.
    In all but a few cases [27], however, such attention mechanisms are used in conjunction with a recurrent network.
    The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building block, computing hidden representations in parallel for all input and output positions.
  • RESEARCH_GAP: A single attention head averages information, which inhibits the model from attending to diverse aspects of the input.
    Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.
    With a single attention head, averaging inhibits this.