FrugalGPT

Full title: FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance Authors: Lingjiao Chen, Matei Zaharia, James Zou (Stanford University) Venue/year: arXiv preprint, May 2023 — arXiv:2305.05176 Links: arXiv · local PDF: research-papers/2305.05176-frugalgpt.pdf

Why this paper matters

FrugalGPT is the founding paper of LLM inference economics as a research problem. Written weeks after GPT-4's release, it made three moves that stuck: (1) it framed LLM APIs as a heterogeneous marketplace — 12 commercial APIs from 5 providers whose prices differ by up to two orders of magnitude (Table 1); (2) it organized the cost-reduction design space into three reusable strategies — prompt adaptation, LLM approximation, and LLM cascade (§3); and (3) it demonstrated a concrete, learned cascade that matches GPT-4's accuracy at up to 98% lower cost, or beats it by up to 4% accuracy at equal cost (abstract, Table 3). Every model router, cascade product, and "small model first, escalate on doubt" serving pattern in today's agent stacks descends from the optimization problem this paper wrote down. It reframed "which LLM should I call?" from a fixed choice into a per-query, budget-constrained decision.

The problem

Using the best LLM for everything is prohibitively expensive: the paper cites estimates of ChatGPT costing over $700K/day to operate, and works a case study where a 15,000-customer small business pays ~$21.2K/month running customer service on GPT-4 (§1, §2). Meanwhile the API market is wildly heterogeneous — GPT-4 cost $30 per 10M input tokens vs $0.2 for GPT-J on Textsynth (§1, Table 1) — and cheap models are not uniformly worse: on HEADLINES, GPT-J answers correctly on ~6% of the cases where GPT-4 is wrong (Figure 4). Existing model-ensemble and cascade literature (FrugalML etc.) assumed predictive tasks with a fixed, known label set and white-box access; LLM APIs are black boxes emitting free text, and both prompt choice and API choice affect cost and quality (§1, Related Works). Nobody had formalized budget-aware LLM API usage: maximize task performance subject to an average per-query cost constraint.

The mechanism

The marketplace and cost model (§2)

A market of K LLM APIs, {f_i}, i = 1..K, each mapping a prompt p to an answer. Cost of calling API i decomposes into three published components:

c_i(p) = c~_{i,2} * ||f_i(p)||  +  c~_{i,1} * ||p||  +  c~_{i,0}
         (output tokens)           (input tokens)       (fixed fee per request)

The paper grounds the stakes with a worked example (§2): a small business running GPT-4 customer service for 15,000 customers, each asking three questions twice a week (360,000 queries/month), with ~1,800 prompt tokens and ~80 answer tokens per query, pays 360 × ($0.03 × 1800 + $0.06 × 80) ≈ $21.2K per month.

The problem statement: choose a strategy s (prompts, APIs, aggregation) to

max  E_{(q,a)}[ r(a, â(s,q)) ]      s.t.   E_{(q,a)}[ c(s,q) ] <= b

where q is a query, a the correct answer, â(s,q) the strategy's answer, r the reward (answer quality), c(s,q) the incurred cost, and b the user's budget (§2, "Problem statement"). The search space of s is "vast, encompassing factors such as which prompts to use, which LLM APIs to employ, and how to aggregate their responses" — the cascade of §3 is one deliberately simple point in it.

The three cost-reduction strategies (§3, Figure 2)

1. Prompt adaptation — shrink what you send: prompt selection (keep only a subset of few-shot examples) and query concatenation (send one shared prompt with many queries batched into a single request, so the prompt is processed once).
2. LLM approximation — imitate the expensive API: completion cache (store answers, serve similar queries from the cache) and model fine-tuning (collect the expensive model's outputs, fine-tune a cheap model on them — also cuts latency since the fine-tuned model needs no long prompt).
3. LLM cascade — the strategy FrugalGPT actually implements: call APIs in a learned order, stop as soon as an answer looks reliable. Cheap models absorb easy queries; the expensive model is reserved for queries the scorer distrusts.

The cascade's two learned components (§3, Strategy 3)

• Generation scoring function g(q, a) : Q × A → [0,1] — a reliability score for answer a to query q. Instantiated as a simple regression model (a DistilBERT tailored to regression in the experiments, §4) trained to predict whether a generation is correct. Note that it's much smaller and cheaper than every LLM in the market.
• LLM router — chooses the list L ∈ [K]^m of m APIs to try in order, plus a threshold vector τ = (τ_1, …, τ_m). At query time: invoke f_{L_i}(q), compute g(q, f_{L_i}(q)); return the answer if the score exceeds τ_i, otherwise escalate to the next API.

The five concrete tricks of Figure 2, mapped to strategies — note how little of the vision the paper actually benchmarks (only the last):

Compositions (§3)

The three strategies compose: joint prompt and LLM selection searches, per query, for the smallest prompt and cheapest API that reach satisfactory quality; another composition searches across both existing APIs and fine-tuned models. The paper flags the trade honestly — composing approaches "also increases the computational costs for training," opening a three-way trade-off between query cost, task performance, and optimization compute (§3, "Compositions"). FrugalGPT-as-evaluated implements only the cascade; the compositions are the announced research agenda.

The algorithm

Inference-time cascade (reconstructed from §3, Figure 2e)

# Learned offline: L = (L_1..L_m)  ordered API list, tau = (tau_1..tau_m) thresholds
def frugalgpt_answer(q):
    for i in 1..m:
        a_i = f_{L_i}(q)                    # call the i-th API in the list
        if g(q, a_i) >= tau_i:              # scorer trusts this answer
            return a_i                      # stop; later (pricier) APIs never called
    return a_m                              # last API's answer is the fallback

The training-time optimization (§3, rendered verbatim in structure)

max_{L, tau}   E[ r(a, f_{L_z}(q)) ]

s.t.   E[ sum_{i=1}^{z}  c~_{L_i,2} ||f_{L_i}(q)|| + c~_{L_i,1} ||q|| + c~_{L_i,0} ]  <=  b

where  z = arg min_i  { i :  g(q, f_{L_i}(q)) >= tau_i }

Every symbol:

Reading the program:

• Objective — expected reward of the answer actually returned (the one at stopping position z), over the query distribution.
• Constraint — expected cumulative cost of all APIs invoked up to and including position z (you pay for every model you tried, not just the winner), bounded by budget b. Each term is the three-part cost model applied to API L_i.
• Threshold semantics — low τ_i = accept eagerly (cheap, riskier); high τ_i = escalate often (costly, safer). The learned HEADLINES thresholds (0.96 then 0.37) encode "distrust the cheapest model unless it's nearly certain; be lenient with the mid-tier one."

This is a mixed-integer program (discrete L, continuous τ) and "computationally expensive to solve" naively; the paper's specialized optimizer (i) prunes candidate lists whose LLMs disagree too little (a cascade of near-clones can't help — complementarity is what pays), and (ii) approximates the objective by interpolating it within a few samples rather than evaluating every (L, τ) on all training data (§3).

Worked instance (HEADLINES, budget $6.5 = one fifth of GPT-4's cost, §4 "A Case Study", Figure 3): the learned cascade is GPT-J → J1-L → GPT-4 with thresholds 0.96 and 0.37 — accept GPT-J's answer only if the scorer is very confident (≥0.96), accept J1-L's above 0.37, otherwise pay for GPT-4. Result: accuracy 0.872 vs GPT-4's 0.857, at cost $6.5 vs $33.1 — 80% cheaper and 1.5% more accurate, because the cascade routes around queries GPT-4 gets wrong but cheaper models get right.

Results that matter

Datasets (Table 2): HEADLINES (financial news gold-price trend, 10,000 examples), OVERRULING (legal overruling detection, 2,400), COQA (reading comprehension adapted to direct QA, 7,982). Cascade length fixed at 3 (§4).

Two structural observations from §4 ("Performance and Cost Trade-offs") worth keeping:

• Cost rankings are task-dependent. J1 is the second-most-expensive API on HEADLINES but GPT-3 takes that spot on OVERRULING and COQA — because J1 charges only for output tokens while GPT-3 charges for both, so relative cost flips with the prompt/answer length ratio.
• Price does not order quality. J1 costs more than GPT-3 on HEADLINES yet performs worse — "underscor[ing] the importance of aptly selecting LLM APIs, even in the absence of budget constraints."
• MPI, precisely: the maximum performance improvement of LLM A over LLM B is the probability that A is correct where B is wrong (Figure 4 caption) — an upper bound on what adding A to a B-anchored cascade can gain.

Limitations & how to defend the findings

• "The scorer needs labeled training data — that's hidden cost." Conceded in §5: the cascade needs labeled examples, and learning it "requires resources." The defense, also §5: it is a one-time upfront cost, amortized whenever the query stream is larger than the training set — the regime the paper targets (high-throughput applications, §1).
• "It only works in-distribution." True and stated: training examples should be "from the same or similar distribution as the test examples" (§5). Nothing in the paper measures distribution shift; a production system needs drift monitoring around g(q,a). This is the paper's biggest practical gap.
• "These are three classification-ish tasks, not open-ended generation." Fair. r(·,·) needs a checkable notion of correctness, and all three datasets provide one. The framework (§2) is stated for general natural-language query answering, but the evidence is classification/QA-shaped. Open-ended generation scoring is future work.
• "Latency: a 3-deep cascade can be 3 API calls." Acknowledged as unmodeled — §5 lists latency, fairness, privacy, and environmental impact as factors real applications need that the optimization ignores. (Fine-tuning-based approximation, §3, is the latency-friendly alternative the paper sketches but doesn't quantify.)
• "Redundant calls waste money when all models agree." The paper shows this failure honestly: in the third COQA example of Figure 5, all three LLMs give the same correct answer but low scores force the cascade through all of them; "identifying how to avoid such cases remains an open problem" (§4).
• "Prices moved; the exact numbers are stale." Prices are from March 2023 (Table 1 caption). The structure — two-orders-of-magnitude spread, per-token asymmetries (J1 charges only for output; GPT-3 for both, §4) — is what the method exploits, and that heterogeneity persists.

Connections

• Router vs cascade economics: a router picks one model per query before calling (one call, needs to predict quality unseen); FrugalGPT's cascade decides after seeing each answer (more calls, but decisions are grounded in an actual generation scored by g). Modern serving stacks use both; this paper is the cascade pole of that trade-off.
• Complementarity is the profit source: Figure 4's MPI matrices are the paper's deepest observation — cheap and expensive LLMs make different mistakes, so a cascade can exceed the best single model, not just approximate it. Same diversity phenomenon that powers self-consistency and ensembles.
• Let's Verify (sibling page): both papers hinge on a learned scorer over generations. The PRM asks "which of my N samples is right?" (reliability); g(q,a) asks "is this answer good enough to stop paying?" (economics). A cascade is best-of-N sequentialized under a budget.
• τ-bench (sibling page): τ-bench's cost analysis (~$0.38+$0.23 per agent task) shows agent workloads are exactly the high-volume regime FrugalGPT targets; per-step model cascading inside agent loops is the natural composition.
• LLM approximation ≈ distillation + caching: the completion cache prefigures today's semantic caches; fine-tuning on the expensive model's outputs is knowledge distillation reframed as a billing optimization.
• Prompt adaptation anticipates prompt-compression and batched-inference lines of work — cost scales with prompt length, so the prompt itself is an optimization variable (§3, Strategy 1).

Reading guide

• Read first: Abstract → §1 (market framing + Figure 1) → §3 Strategy 3 (the cascade + optimization — the technical core) → §4 "A Case Study" with Figure 3.
• Then: Table 1 (the price table that motivates everything), Table 3 (headline savings), Figure 4 (MPI/complementarity — the why it works), §5 (unusually honest limitations).
• Skimmable: §3 Strategies 1–2 (prompt adaptation, approximation — described but never evaluated), Related Works in §1, Figure 2 (illustrations of all five tricks).
• The one figure to study: Figure 5 — accuracy-vs-cost frontiers on all three datasets with per-query cascade traces on the right. It shows the red FrugalGPT curve dominating every individual LLM point, and concrete queries where GPT-J's accepted answer beats GPT-4's — the whole thesis in one panel.