Public demo

Dense Passage Retrieval for Open-Domain Question Answering

Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih (2020)

Dense Passage Retrieval uses dual encoders to retrieve relevant passages for open-domain question answering without relying only on sparse lexical matching.

Why it matters

DPR made dense vector retrieval practical for question answering pipelines and helped establish the retrieval layer that later RAG systems depend on.

Highlights

  • [method] DPR encodes questions and passages separately, then retrieves by vector similarity. — why: this dual-encoder setup is the core dense retrieval pattern.
  • [finding] Dense retrieval can outperform traditional sparse retrieval on open-domain QA benchmarks when trained with suitable supervision. — why: it motivates embedding-based search for evidence retrieval.
  • [method] Negative passage selection is important for training useful retrievers. — why: retrieval quality depends heavily on what the model learns to distinguish.
  • [limitation] Dense retrievers can miss exact lexical cues and are sensitive to domain shift. — why: research writing often needs both semantic similarity and precise terminology.

Reader notes

  • DPR is the retrieval backbone idea behind many later RAG systems; connect it forward to retrieval-augmented-generation.
  • For SAV, dense retrieval alone is not enough: the product needs passage-level traceability, citation metadata, and user verification.
  • Useful citation when explaining why embeddings help find evidence that keyword search may miss.

Extracted evidence examples

  • OBJECTIVE: Dense passage retrieval learns embedding functions to map question-passage pairs into a vector space, enabling efficient retrieval of relevant passages for open-domain question answering.
    In this paper, we address the question: can we train a better dense embedding model using only pairs of questions and passages (or answers), without additional pretraining?
    We focus our research in this work on improving the retrieval component in open-domain QA.
  • RESEARCH_PROBLEM: Effective open-domain QA relies on passage retrieval to narrow search space, with dense encodings offering flexible, learnable representations superior to sparse methods.
    Open-domain question answering (QA) (Voorhees, 1999) is a task that answers factoid questions using a large collection of documents.
    While early QA systems are often complicated and consist of multiple components (Ferrucci (2012); Moldovan et al. (2003), inter alia), the advances of reading comprehension models suggest a much simplified two-stage framework: (1) a context retriever first selects a small subset of passages where some of them contain the answer to the question, and then (2)...
  • RESEARCH_GAP: Dense retrieval methods, while capturing semantics, often underperform sparse methods for QA without careful negative example selection and encoder fine-tuning, and may miss rare salient phrases.
    A term-based system would have difficulty retrieving such a context, while a dense retrieval system would be able to better match "bad guy" with "villain" and fetch the correct context.
    Dense retrieval methods have thus never be shown to outperform TF-IDF/BM25 for opendomain QA before ORQA (Lee et al., 2019), which proposes a sophisticated inverse cloze task (ICT) objective, predicting the blocks that contain the masked sentence, for additional pretraining.
  • RESEARCH_GAP: The time required for building an index for dense vectors is significantly longer than for traditional sparse methods.
    On the other hand, the time required for building an index for dense vectors is much longer.
    Computing dense embeddings on 21-million passages is resource intensive, but can be easily parallelized, taking roughly 8.8 hours on 8 GPUs.
  • RESEARCH_GAP: The SQuAD dataset introduces unwanted bias in passage retrieval due to its limitation to a small set of Wikipedia documents.
    When training with mul-8 SQuAD is limited to a small set of Wikipedia documents and thus introduces unwanted bias.
  • CONTRIBUTION: Dense retrieval, when fine-tuned on question-passage pairs, significantly improves open-domain QA accuracy, outperforming sparse methods and rendering additional pretraining unnecessary.
    First, we demonstrate that with the proper training setup, simply fine-tuning the question and passage encoders on existing question-passage pairs is sufficient to greatly outperform BM25.
    Our empirical results also suggest that additional pretraining may not be needed.
  • METHODOLOGY: Dual-encoder models learn question and passage encoders, optimizing for dot-product similarity with in-batch negatives for efficient, low-latency retrieval.
    With special in-memory data structures and indexing schemes, retrieval can be done efficiently using maximum inner product search (MIPS) algorithms (e.g., Shrivastava and Li (2014); Guo et al. (2016)).
    By leveraging the now standard BERT pretrained model (Devlin et al., 2019) and a dual-encoder architecture (Bromley et al., 1994), we focus on developing the right training scheme using a relatively small number of question and passage pairs.
  • METHODOLOGY: Articles are processed to extract clean text, removing semi-structured data like tables and infoboxes.
    We first apply the pre-processing code released in DrQA (Chen et al., 2017) to extract the clean, text-portion of articles from the Wikipedia dump.
    This step removes semi-structured data, such as tables, infoboxes, lists, as well as the disambiguation pages.