DPR

Full title: Dense Passage Retrieval for Open-Domain Question Answering Authors: Vladimir Karpukhin*, Barlas Oğuz*, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih (Facebook AI, University of Washington, Princeton) Venue/Year: EMNLP 2020 arXiv: 2004.04906 Local PDF: /Users/moscalej/Documents/Onep/research-papers/2004.04906-dpr.pdf

Why this paper matters

DPR is the paper that made dense embedding retrieval the default over keyword search. Before it, open-domain QA systems retrieved with TF-IDF/BM25, and the one dense predecessor (ORQA, 2019) needed an expensive inverse-cloze pretraining task to beat sparse retrieval. DPR showed that none of that is necessary: two plain BERT-base encoders, fine-tuned on existing question–passage pairs with the right contrastive objective (in-batch negatives plus one BM25 hard negative), beat BM25 by 9–19 points absolute in top-20 retrieval accuracy and set new end-to-end QA state of the art on four of five benchmarks (abstract, §1). Every modern embedding-based RAG stack — the "retriever" half of retrieval-augmented agents — is a descendant of this recipe: bi-encoder, dot-product similarity, ANN index, hard-negative mining.

The problem

Open-domain QA is a two-stage pipeline: a retriever narrows a huge corpus to a few passages, then a reader extracts the answer (§1). The retriever was the bottleneck, and the incumbent retrievers were sparse:

• TF-IDF/BM25 can't bridge vocabulary gaps. The paper's example: "Who is the bad guy in lord of the rings?" should match a passage saying Sala Baker "is best known for portraying the villain Sauron" — a term-matcher misses bad guy ↔ villain; a learned dense space can align them (§1).
• Dense retrieval was believed to need massive pretraining. ORQA's inverse cloze task (ICT) was computationally intensive, its pseudo-question surrogates were of unclear quality, and its context encoder was never fine-tuned on real question–passage pairs, leaving representations suboptimal (§1).

The open question the paper poses verbatim: can we train a better dense embedding model using only pairs of questions and passages, without additional pretraining? (§1).

The mechanism

The dual encoder (§3.1)

Two independent BERT-base (uncased) networks: a passage encoder E_P(·) and a question encoder E_Q(·). Each maps its text to the [CLS] token representation, so d = 768. Similarity is a plain dot product:

sim(q, p) = E_Q(q)^T · E_P(p)        (Eq. 1)

Why a dot product and not cross-attention? Because the similarity must be decomposable: passage vectors are precomputed offline for all 21M passages and indexed with FAISS, so at query time only one question encoding plus a maximum-inner-product search is needed. The ablation (§5.2, Appendix B) shows L2 distance performs comparably and cosine worse — so the simple choice stands, and the effort goes into learning better encoders instead.

Corpus preparation (§4.1)

English Wikipedia (Dec 20, 2018 dump), cleaned with DrQA's preprocessor, split into non-overlapping 100-word passages — 21,015,324 in total. Each passage is prepended with its article title and a [SEP] token. Fixed-length blocks worked better than natural paragraphs in their trials (footnote 3).

Training as metric learning (§3.2)

The goal is a vector space where a question and its relevant passages have higher dot product than irrelevant ones. Each training instance is one question, one positive passage, and n negatives; the loss is the negative log-likelihood of the positive (Eq. 2 — rendered in the algorithm section below).

Negative selection is the paper's key insight. Positives come with the dataset; negatives must be picked from ~21M candidates, and the choice "is often overlooked but could be decisive" (§3.2). Three types are compared:

1. Random — any corpus passage.
2. BM25 — top BM25 passages that do not contain the answer string: lexically similar decoys, i.e., hard negatives.
3. Gold — positive passages belonging to other questions.

The best configuration: gold passages of the other questions in the same mini-batch as negatives, plus one BM25 hard negative shared across the batch (§3.2, Table 3).

In-batch negatives: with B questions and B positive passages in a batch, compute S = Q P^T, a B×B score matrix. Row i treats passage i as positive and the other B−1 as negatives — B² training pairs from B forward passes, essentially free reuse of computation. Accuracy "consistently improves as the batch size grows" (§5.2).

What "retrieval accuracy" means here (§2)

A retriever R(q, C) returns a filter set of k passages from corpus C. Top-k retrieval accuracy is the fraction of questions for which the filter set contains a span that answers the question — the paper's primary metric (k ∈ {20, 100} in Table 2), chosen because the retriever's job is to hand the reader a small candidate set that still contains the answer.

Training setup (§5)

• Batch size 128, one extra BM25 hard negative per question (shared across the batch).
• Up to 40 epochs on large datasets (NQ, TriviaQA, SQuAD); 100 on small ones (TREC, WQ).
• Adam at learning rate 10⁻⁵, linear scheduling with warm-up, dropout 0.1.
• Multi-dataset variant ("Multi"): one encoder trained on the union of all datasets except SQuAD (§5).
• Hybrid baseline: rerank the union of top-2000 from each system by BM25(q,p) + λ·sim(q,p), λ = 1.1 (§5).

Inference (§3.1, §5.4)

All passage vectors go into a FAISS HNSW index. Throughput: 995 questions/second returning top-100 passages on a CPU server, vs 23.7 q/s per thread for BM25/Lucene (§5.4). The trade: index construction — 8.8 hours on 8 GPUs to embed 21M passages plus 8.5 hours to build the FAISS index, vs ~30 minutes for a Lucene inverted index.

The algorithm

# ---- Training ----
Input: dataset D = {(q_i, p_i+)} of question/positive-passage pairs; BM25 index over corpus
for each mini-batch of B pairs {(q_1, p_1+), ..., (q_B, p_B+)}:
    for each q_i: fetch one hard negative h_i = top BM25 passage that does NOT contain the answer
    Q = [E_Q(q_1); ...; E_Q(q_B)]                        # B x 768 question matrix
    P = [E_P(p_1+); ...; E_P(p_B+); E_P(h_1); ...; E_P(h_B)]   # 2B x 768 passage matrix
    S = Q @ P.T                                          # B x 2B score matrix (dot products)
    loss = mean_i [ -log( exp(S[i,i]) / sum_j exp(S[i,j]) ) ]   # softmax CE, column i is the positive
    update E_Q, E_P by gradient descent

# ---- Indexing (offline) ----
for every passage p in corpus (21M): v_p = E_P(p)
build FAISS index over {v_p}

# ---- Retrieval (online) ----
v_q = E_Q(q)
return top-k passages by inner product v_q · v_p        # k = 20..100

Line by line:

• The loss (Eq. 2):

  L(q_i, p_i⁺, p_{i,1}⁻, …, p_{i,n}⁻) = −log [ e^{sim(q_i, p_i⁺)} / ( e^{sim(q_i, p_i⁺)} + Σ_{j=1}^{n} e^{sim(q_i, p_{i,j}⁻)} ) ]

  Every symbol, spelled out:

  Symbol	Meaning
  q_i	the i-th training question
  p_i⁺	its annotated relevant (positive) passage
  p_{i,j}⁻	the j-th irrelevant (negative) passage for q_i; n of them per question
  sim(·,·)	the dot product E_Q(q)ᵀE_P(p) of Eq. 1
  E_Q, E_P	the two BERT-base encoders (768-d [CLS] outputs)
  B	mini-batch size (128 in the main experiments); under in-batch training, n = B−1 gold negatives + 1 BM25 negative

  This is softmax cross-entropy over one positive and n negatives — the InfoNCE/contrastive form. Minimizing it pushes sim(q, p⁺) up relative to every negative, shaping a space where relevance = inner product.

• Why in-batch negatives work: the batch already contains B passage encodings; scoring every question against every passage (the S matrix) turns B examples into B×(B−1) extra negative pairs at no additional encoding cost. Table 3 shows the same 7 gold negatives jump from 42.6 to 51.1 top-5 accuracy on NQ dev when switched from per-question sampling to in-batch, and scaling in-batch size 7→31→127 climbs 51.1→52.1→55.8.

• Why exactly one BM25 negative: it is a passage that looks right to a term-matcher but is wrong — the most informative kind of error to train against. Adding it to the 127 in-batch negatives lifts top-5 from 55.8 to 65.8 (Table 3, "G.+BM25(1)"); adding a second BM25 negative does not help further (64.5).

• Positives when the dataset gives only Q/A pairs (TREC, WQ, TriviaQA): take the highest-ranked BM25 passage that contains the answer; discard the question if none of the top 100 does (§4.2). Appendix A shows this distant supervision costs only ~1 point vs gold contexts.

Results that matter

The five benchmarks (Table 1; "Train" = questions used for training DPR after filtering / Test):

• Natural Questions (NQ): 58,880 / 3,610 — real Google queries, answers as Wikipedia spans.
• TriviaQA: 60,413 / 11,313 — trivia questions scraped from the Web.
• WebQuestions (WQ): 2,474 / 2,032 — Google Suggest questions with Freebase entity answers.
• CuratedTREC (TREC): 1,125 / 694 — TREC QA-track questions, unstructured-corpus QA.
• SQuAD v1.1: 70,096 / 10,570 — reading-comprehension questions written while viewing the passage.

Limitations & how to defend the findings

• "It loses to BM25 on SQuAD — so dense retrieval isn't strictly better." True, and the paper says why (§5.1): SQuAD annotators wrote questions while looking at the passage, so lexical overlap is unusually high (BM25's home turf), and the data covers only 500+ Wikipedia articles, an extremely biased distribution. The honest generalization: dense wins when the query paraphrases the evidence; sparse wins on high-overlap, keyphrase-style queries. Appendix C's "Thoros of Myr" example shows DPR missing a salient rare phrase that BM25 nails — this motivates today's hybrid search.
• "Maybe it's just the training data volume." Figure 1 rebuts this: with only 1k pairs DPR already beats BM25 on NQ dev; more data helps but is not the enabling factor — the pretrained LM plus the contrastive setup is.
• "Is the gain from the architecture or the negatives?" Table 3 isolates it: same model, negatives varied. Without in-batch training the negative type barely matters (top-20 ≈ 63); in-batch negatives add ~10 points top-5; the single BM25 hard negative adds ~10 more. The recipe, not the network, is the contribution.
• "Dot product is too crude a similarity." §5.2/Appendix B: L2 performs comparably, cosine worse, triplet loss ≈ NLL. More expressive cross-attention scoring is better per pair but non-decomposable, so it cannot be pre-indexed — DPR deliberately spends capacity in the encoders, and leaves cross-attention to the reader/reranker stage (§6.1).
• "Fixed 100-word chunks are arbitrary." Acknowledged (footnote 3): passage granularity is a function of both retriever and reader; fixed blocks simply beat natural paragraphs in their trials.
• "Index freshness." Adding or changing documents requires re-embedding; the 8.8 GPU-hour embedding cost vs Lucene's 30 minutes (§5.4) is the operational price of dense retrieval.
• "Does it transfer without fine-tuning?" §5.2 cross-dataset: NQ-trained DPR applied to WebQuestions/TREC loses only 3–5 points vs in-domain fine-tuning (69.9/86.3 vs 75.0/89.1 top-20) and still crushes BM25 (55.0/70.9).

Connections

• InfoNCE / contrastive learning: Eq. 2 is the InfoNCE objective specialized to retrieval; in-batch negatives are the same trick used in CLIP-style multimodal training. DPR is the canonical text-retrieval instance.
• Modern embedding models: sentence-transformer / OpenAI-style embedding APIs used in agent memory stores follow DPR's bi-encoder + MIPS blueprint; hard-negative mining (ANCE, which the paper cites in §7 as its own successor) iterates DPR's BM25-negative idea using the model itself.
• Hybrid search: the SQuAD loss and the Table 7 keyphrase failure are the empirical justification for BM25+dense fusion (the paper's own BM25+DPR linear combination, λ=1.1, §5) — today's hybrid retrieval in RAG pipelines.
• RAG: §7 notes DPR "can be combined with generation models such as BART and T5" — that line points directly at Lewis et al.'s RAG, which uses this exact retriever.
• HippoRAG (sibling page): uses retrieval encoders like DPR as one component but argues single-vector similarity cannot do multi-hop association — its PPR-over-KG mechanism is a response to DPR's single-step limits.
• Reranking stage: DPR's reader (Eqs. 3–5) doubles as a cross-attention reranker over the top-k — the retrieve-then-rerank pattern now standard in production RAG.

Reading guide

• Read first: §3 (model + training, 2 pages — the whole contribution) → Table 3 (§5.2) → Table 2 (§5.1).
• The one table to study: Table 3 — it decomposes the recipe (negative type × in-batch × BM25 hard negative) and is where the method actually lives.
• Then: §5.1 main results incl. the SQuAD caveat; §5.4 for the latency/index-cost trade-off; §6/Table 4 for end-to-end QA.
• Skimmable: §2 (background/task formalism), §4 (dataset details), §7 (related work), Appendix D (joint training — a null result: 39.8 EM, no better than the pipeline).
• Worth a detour: Appendix C / Table 7 — two concrete examples of where DPR beats BM25 (semantic match, "body of water" → Irish Sea) and where it fails (rare salient phrase "Thoros of Myr").