MT-Bench

Full title: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Authors: Lianmin Zheng*, Wei-Lin Chiang*, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica (UC Berkeley, UC San Diego, CMU, Stanford, MBZUAI) Venue/Year: NeurIPS 2023, Datasets and Benchmarks Track (arXiv v4, Dec 2023) arXiv: 2306.05685 Local PDF: /Users/moscalej/Documents/Onep/research-papers/2306.05685-mt-bench.pdf

Why this paper matters

This is the paper that legitimized "LLM-as-a-judge" — the practice of using a strong LLM (GPT-4) to grade other models' open-ended outputs — by actually measuring it against human judgment instead of assuming it. The LMSYS team built two human-preference benchmarks (MT-Bench, 80 curated multi-turn questions; Chatbot Arena, ~30K crowdsourced battle votes), collected ~3K expert votes, and showed that GPT-4's verdicts agree with human experts over 80% of the time — the same level as humans agree with each other (abstract, §4.2). Just as importantly, it named and quantified the failure modes — position bias, verbosity bias, self-enhancement bias, weak math grading — and proposed the mitigations (position swapping, reference-guided grading) that are now standard practice. Every automated eval pipeline in agentic systems that scores outputs with a model instead of a human — including Mem0's "J" metric on this shelf — inherits both this method and its caveats.

The problem

RLHF-aligned chat models are "strongly preferred by human users over the original, unaligned models," yet MMLU/HELM-style benchmarks "cannot effectively tell the difference between these aligned models and the base models" (§1) — Figure 1's demonstration: base LLaMA-13B scores competitively on MMLU but gives a repetitive, unhelpful answer humans reject. Existing benchmarks measure core capability on closed-ended questions with short answers (§2.1); none measure alignment with human preference in open-ended, multi-turn dialogue. Human evaluation is the gold standard but "exceptionally slow and costly" (§1); rule-based programs can't grade reference-free open-ended answers, and similarity metrics like ROUGE/BLEU "are also ineffective for these questions" (§3). The gap: a scalable, automated proxy for human preference — and a systematic study of whether that proxy can be trusted, which "has not been" done before this paper (§1).

The mechanism

The two benchmarks (§2.2–2.3)

• MT-Bench: 80 hand-designed questions, each with two turns (the second turn stress-tests instruction following), across 8 categories: writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), knowledge II (humanities/social science) — 10 questions per category (§2.2). The flavor of the two-turn design (Table 1):

  Category	1st turn	2nd turn
  Writing	"Compose an engaging travel blog post about a recent trip to Hawaii…"	"Rewrite your previous response. Start every sentence with the letter A."
  Math	"Given that f(x) = 4x³ − 9x − 14, find the value of f(2)."	"Find x such that f(x) = 0."
  Knowledge	"Provide insights into the correlation between economic indicators such as GDP, inflation…"	"Now, explain them again like I'm five."

• Chatbot Arena: a crowdsourcing platform of anonymous side-by-side battles; users chat with two hidden models simultaneously and vote, identities revealed only after voting. One month of operation yielded ~30K votes in the wild, with no predefined questions (§2.3).

The three judge protocols (§3.1)

1. Pairwise comparison — judge sees the question and two answers; outputs A wins / B wins / tie. Most sensitive, but comparisons grow quadratically in the number of models.
2. Single-answer grading — judge assigns an absolute score to one answer (MT-Bench uses a 1–10 scale per turn, §5). Scales linearly; "may be unable to discern subtle differences between specific pairs," and absolute scores can drift if the judge model changes (§3.1).
3. Reference-guided grading — the prompt additionally includes a reference solution (used for math, §3.4).

Trade-offs the paper states explicitly (§3.1): pairwise is the most discriminative but the number of comparisons "grows quadratically" with the number of models; single-answer grading scales linearly and is more stable across re-runs of a leaderboard, but "may be unable to discern subtle differences between specific pairs" and its absolute scores "are likely to fluctuate more than relative pairwise results if the judge model changes."

For multi-turn questions, the judge must see both full conversations in a single prompt; splitting turns into separate prompts makes the judge misattribute earlier context — GPT-4 was observed blaming Assistant A for text it never produced (§3.5, Figure 16).

The agreement methodology (§4.1)

Judges are validated the only way that is epistemically sound: against humans, with human-vs-human agreement as the ceiling.

• Setup: all 6 models (GPT-4, GPT-3.5, Claude-V1, Vicuna-13B, Alpaca-13B, LLaMA-13B) answer all 80 questions; 58 expert-level labelers (mostly graduate students) cast ~3K votes on pairs; LLM judges evaluate all pairs. On Arena, 3K single-turn votes are sampled from 30K (2,114 unique voter IPs).
• Agreement metric: "the probability of randomly selected individuals (but not identical) of each type agreeing on a randomly selected question" (§4.1).
• Two accounting regimes: S1 counts ties and position-inconsistent votes (random baseline 33%); S2 keeps only non-tie votes (random baseline 50%) — always check which regime a quoted agreement number uses (Table 5).
• Comparability: single-answer scores are converted into pairwise outcomes so all judge types share one metric (Table 5 caption).

The three bias mechanisms and their mitigations (§3.3–3.4)

1. Position bias — favoring an answer because of where it sits in the prompt. Measured with a controlled probe: two near-identical answers (GPT-3.5 called twice at temperature 0.7), judged in both orders. Consistency (same verdict after swap): Claude-v1 23.8%, GPT-3.5 46.2%, GPT-4 65.0% — with Claude-v1 favoring the first position 75.0% of the time (Table 2). Renaming assistants to "Assistant A/B" variants shows Claude-v1 also carries a name bias (biased toward "Assistant A"). Mitigation (§3.4): call the judge twice with swapped order; declare a winner only if both orders agree, else tie (the conservative rule the paper adopts). Few-shot examples raise GPT-4's consistency 65.0% → 77.5% but cost 4× and may introduce new biases, so zero-shot stays the default.

2. Verbosity bias — favoring longer answers regardless of quality. Probed with a "repetitive list" attack: take 23 MT-Bench answers containing numbered lists, have GPT-4 rephrase the list without adding information, and prepend it to the original. Attack success (judge prefers the padded answer): Claude-v1 91.3%, GPT-3.5 91.3%, GPT-4 8.7% (Table 3). Calibration check: all judges correctly tie truly identical answers. Mitigation: none fully solves it in-paper; judge choice matters (GPT-4 "defends significantly better", §3.3).

3. Self-enhancement bias — a judge favoring its own outputs. The evidence is statistical and inconclusive: as a judge, GPT-4 rates GPT-4 answers with a 10% higher win rate than humans do, Claude-v1 favors itself by 25%, but GPT-3.5 does not favor itself, and the authors state the data "cannot determine whether the models exhibit a self-enhancement bias" — a controlled study would require restyling an answer without changing its quality (§3.3). Mitigation: treat as an open risk; avoid judging a model with itself where it matters.

Plus a capability limit: grading math/reasoning. GPT-4 misgrades elementary math it can solve on its own because the shown answers mislead it (§3.3, Figure 13). Chain-of-thought judging (answer independently first, then grade) only drops the failure rate from 14/20 to 6/20 — the judge often reproduces the same error the answer made. Reference-guided grading — generate the judge's own solution separately, then insert it as reference — cuts failures to 3/20 (70% → 15%, Table 4, §3.4).

The algorithm

# ---- Judge protocols ----
def pairwise_judge(q, ans_A, ans_B, J):                 # §3.1, prompt Fig. 5
    v1 = J(prompt(q, ans_A, ans_B))                     # verdict in {A, B, tie}
    v2 = J(prompt(q, ans_B, ans_A))                     # swapped order (§3.4)
    return v1 if v1 == flip(v2) else "tie"              # conservative de-biasing

def single_grade(q, ans, J):                            # §3.1, prompt Fig. 6
    return J.score(prompt(q, ans), scale=1..10)         # per turn; MT-Bench score = mean over 160 turns

def reference_judge(q, ans_A, ans_B, J):                # §3.4, math/reasoning
    ref = J.solve(q)                                    # judge solves independently
    return pairwise_judge(q, ans_A, ans_B, J | ref)     # reference inserted in prompt

# ---- Validation: agreement between judge types ----
# votes_X[q, (m1, m2)] = verdict of a judge of type X (LLM or human)
def agreement(votes_X, votes_Y, setup):
    pairs = all (question, model-pair) events with votes from both types
    if setup == "S2": drop events where either vote is a tie
    return P[ vX == vY ] over random (event, individual-X, individual-Y) draws   # §4.1
# accept the LLM judge iff agreement(LLM, human) >= agreement(human, human)

Walkthrough:

• pairwise_judge: the judge prompt asks for a comparison with an explanation; the both-orders call is the position-bias fix — a win is only a win if it survives the swap. This doubles cost but is what the paper uses at scale (§3.4).
• single_grade: the scalable protocol (linear in models). MT-Bench's leaderboard number is the average score over 160 = 80 × 2 turns (§5). The paper's finding that single grading "matches both pairwise GPT-4 and human preferences very well" (§4.2) is what makes cheap eval pipelines defensible.
• reference_judge: the key subtlety is why CoT is not enough — "LLM makes exactly the same mistake as the given answers in its problem-solving process" when it can see them (§3.4); solving before seeing the answers restores independence.
• agreement: the definition compares individuals, not majorities — so the human-human number (81% on MT-Bench S2, Table 5) is the honest ceiling, and GPT-4's 85% vs humans genuinely means "GPT-4 agrees with a random expert more than two random experts agree with each other."

Results that matter

What the benchmark separates, per category (win rate vs all others, GPT-4 judging; Table 7):

The gap between Vicuna-13B and GPT-3.5/4 concentrates in reasoning, math, and coding (§4.3) — the categories MT-Bench was "carefully constructed" to include precisely so that preference scores would not reduce to style (§1).

Limitations & how to defend the findings

• "85% agreement just means the judge is as noisy as humans." Partly — but that is exactly the claim: LLM-as-a-judge is a proxy for human preference, and it cannot exceed the coherence of the thing it proxies. Figure 2 sharpens it: agreement approaches 100% when models are clearly separated and drops to ~70% for close pairs — so use LLM judges to rank, and distrust small margins.
• "The judge can be gamed by style." Confirmed in-paper: the repetitive-list attack fools Claude-v1 and GPT-3.5 91.3% of the time (Table 3), and §5 shows a model fine-tuned on 3K high-quality dialogues jumps in GPT-4 score without gaining MMLU capability. The paper's own defense is the hybrid framework (§5): pair preference benchmarks with capability benchmarks; no single number determines model quality.
• "Self-enhancement bias invalidates GPT-4 judging GPT-4." The paper is honest that its evidence is suggestive, not conclusive (GPT-4 +10%, Claude +25% self-win-rate vs humans, GPT-3.5 none; §3.3). Defense: the headline agreement results are computed against human votes over six models, so the ranking claims don't rest on self-judgment — but eval pipelines should still avoid judge-equals-contestant setups.
• "Verdicts on math can't be trusted." Correct with default prompts (70% failure on adversarial pairs); reference-guided grading reduces it to 15% (Table 4). Any production judge grading objective tasks should generate-then-grade, not grade directly.
• "80 questions is tiny; experts were grad students; Arena voters are self-selected." All true (§4.1, §6): MT-Bench is a controlled small-scale study, Arena is uncontrolled scale — the two are designed to cover each other's weaknesses, and the paper releases all 3K expert votes and 30K conversations for re-analysis.
• "Helpfulness-only." The paper flags it itself: the metric "emphasizes helpfulness but largely neglects safety," and accuracy/relevance/creativity are "combined into a single metric" (§6). Multi-dimensional and safety judging are explicitly left open.

Connections

• LLM-as-judge in eval pipelines: this paper is the methodological ancestor of model-graded evals everywhere — AlpacaEval, Arena-Hard, and the "J" metric in Mem0's LOCOMO evaluation (see the mem0 page, which inherits both the technique and the need for bias caveats).
• Chatbot Arena → Elo leaderboards: §2.3's battle platform became the LMSYS Chatbot Arena leaderboard, today's de-facto public preference ranking; this paper is its founding validation study.
• RLHF and reward models: a pairwise LLM judge is functionally a zero-shot reward model; the biases catalogued here (position, verbosity) are the same artifacts reward-model training must fight, and Appendix F's fine-tuned Vicuna judge prefigures dedicated judge/reward models.
• Agentic system evaluation: multi-turn judging with full-conversation context (§3.5) is the template for grading agent trajectories, where per-step grading loses referential context.
• Benchmark design: the §5 result — dialog fine-tuning raises judge scores without raising MMLU — is the canonical argument for reporting capability and preference scores side by side rather than collapsing them.
• Goodhart risk: the verbosity attack is an early, concrete demonstration of optimizing-the-judge rather than the task — the failure mode any judge-in-the-loop training or selection pipeline must monitor.

Reading guide

• Read first: §3 in full (judge types, the three biases, mitigations — the conceptual core, ~3 pages) → §4.1–4.2 (agreement definition and headline results).
• The one table to study: Table 5 — it contains the entire argument: GPT-4↔human 85% vs human↔human 81%, under both tie treatments, with vote counts.
• Then: Table 2 (position bias — read alongside the swap mitigation in §3.4), Table 4 (why reference-guided grading exists), §5 + Table 8 (the hybrid-evaluation argument and the style-vs-capability caution).
• Skimmable: §1–2 (motivation and benchmark construction), §4.3 (win-rate curves), references.
• Appendix pulls when needed: Figures 5–9 (the actual judge prompts — worth reading once before writing your own), Appendix C (data collection), Appendix D (bias follow-ups, few-shot judge), Appendix F (fine-tuned Vicuna judge).