Public demo

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela (2020)

RAG combines a neural retriever with a sequence generator so generated answers can condition on retrieved documents rather than only model parameters.

Why it matters

This paper made retrieval a first-class part of generation. It is the bridge from pretrained language models to systems that can ground answers in an external corpus, which is the same broad problem SAV tackles for research writing.

Highlights

  • [method] RAG couples a dense retriever with a generator, using retrieved passages as non-parametric memory. — why: this is the defining architecture of retrieval-augmented generation.
  • [claim] Retrieval lets the model access knowledge outside its fixed parameters. — why: this motivates RAG for knowledge-intensive tasks.
  • [finding] The approach improved performance on several open-domain and knowledge-intensive NLP benchmarks. — why: it showed that retrieval can improve generation quality, not just search.
  • [limitation] Retrieved documents are useful only if they are relevant and correctly used by the generator. — why: RAG shifts the trust problem toward retrieval quality and attribution.

Reader notes

  • This is the anchor paper for the RAG theme; read it before self-rag because Self-RAG adds reflection and control on top of the basic retrieval-generation loop.
  • For SAV, the important idea is not just retrieval, but traceability: writers need to see which source passage supports which claim.
  • RAG is useful vocabulary for investors/users, but the landing page should avoid sounding like a generic RAG wrapper.

Extracted evidence examples

  • OBJECTIVE: Integrating hybrid parametric/non-parametric memory and retrieval modules enhances pre-trained generative language models for broader NLP task capabilities.
    Here, we bring hybrid parametric and non-parametric memory to the "workhorse of NLP," i.e. sequence-to-sequence (seq2seq) models.
    Our work aims to expand the space of possible tasks with a single, unified architecture, by learning a retrieval module to augment pre-trained, generative language models.
  • OBJECTIVE: The paper proposes generating Jeopardy questions to evaluate RAG's generation abilities in a non-QA setting, as it is a more demanding task than standard open-domain QA.
    To evaluate RAG's generation abilities in a non-QA setting, we study open-domain question generation.
    Rather than use questions from standard open-domain QA tasks, which typically consist of short, simple questions, we propose the more demanding task of generating Jeopardy questions.
  • OBJECTIVE: We propose two models that marginalize over latent documents differently to produce a distribution over generated text.
    We propose two models that marginalize over the latent documents in different ways to produce a distribution over generated text.
  • RESEARCH_PROBLEM: Generating factual, knowledge-intensive text like Jeopardy questions or classifying claims based on retrieved evidence presents significant reasoning and precision challenges.
    As Jeopardy questions are precise, factual statements, generating Jeopardy questions conditioned on their answer entities constitutes a challenging knowledge-intensive generation task.
    The task requires retrieving evidence from Wikipedia relating to the claim and then reasoning over this evidence to classify whether the claim is true, false, or unverifiable from Wikipedia alone.
  • RESEARCH_PROBLEM: Large language models store knowledge implicitly in parameters, limiting their ability to access, manipulate, and update knowledge, which poses challenges for knowledge-intensive tasks and provenance.
    Pre-trained neural language models have been shown to learn a substantial amount of in-depth knowledge from data [47].
    While this development is exciting, such models do have downsides: They cannot easily expand or revise their memory, can't straightforwardly provide insight into their predictions, and may produce "hallucinations" [38].
  • RESEARCH_PROBLEM: Open-domain question answering is an important real-world application and a common testbed for knowledge-intensive tasks.
    Open-domain question answering (QA) is an important real-world application and common testbed for knowledge-intensive tasks [20].
  • RESEARCH_GAP: Retrieval-augmented models offer dynamic knowledge updates and improved performance on NLP tasks by aggregating multiple sources, unlike static parametric models or task-specific non-parametric systems.
    While this development is exciting, such models do have downsides: They cannot easily expand or revise their memory, can't straightforwardly provide insight into their predictions, and may produce "hallucinations" [38].
    Hybrid models that combine parametric memory with non-parametric (i.e., retrieval-based) memories [20,26,48] can address some of these issues because knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted.
  • RESEARCH_GAP: Open-domain QA's reliance on public development datasets and simple questions hinders evaluation of advanced generation abilities.
    Rather than use questions from standard open-domain QA tasks, which typically consist of short, simple questions, we propose the more demanding task of generating Jeopardy questions.
    The open-domain QA community customarily uses public development datasets as test datasets, as test data for QA datasets is often restricted and dedicated to reading compehension purposes.