Public demo

Foundations

A source-grounded theme page connecting Attention Is All You Need and BERT: Transformer architecture, pretraining, and reusable representations.

Theme overview

This theme covers the architectural base of modern language models: the Transformer block and bidirectional pretraining. attention-is-all-you-need makes attention the core sequence modeling mechanism; bert turns Transformer encoders into reusable language understanding representations.

Takeaways

• The Transformer is the architecture; BERT is one of the key pretraining recipes built on it.
• This theme should be read first because later demo papers depend on Transformer representations.
• For SAV, these papers explain the model substrate, not the evidence-verification workflow itself.

Included papers

• Attention Is All You Need
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Evidence examples

• Attention Is All You Need: The Transformer, a novel architecture solely using attention, abandons recurrence and convolutions for sequence transduction, demonstrating its importance and generalization capabilities.

  In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.

• Attention Is All You Need: The paper considers three desiderata motivating the use of self-attention layers.

  Motivating our use of self-attention we consider three desiderata.

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding: BERT, a deep bidirectional language model, is introduced with ablation studies to understand its component importance and demonstrate its state-of-the-art fine-tuning capabilities for NLP tasks.

  In this paper, we improve the fine-tuning based approaches by proposing BERT: Bidirectional Encoder Representations from Transformers.

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding: Learning widely applicable representations of words has been an active area of research for decades, including non-neural and neural methods.

  Learning widely applicable representations of words has been an active area of research for decades, including non-neural (Brown et al., 1992;Ando and Zhang, 2005;Blitzer et al., 2006) and neural (Mikolov et al., 2013;Pennington et al., 2014) methods.