Toolformer

Full title: Toolformer: Language Models Can Teach Themselves to Use Tools Authors: Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom (Meta AI Research; Dessì also Universitat Pompeu Fabra) Venue/year: arXiv preprint, February 2023 (cs.CL) arXiv: https://arxiv.org/abs/2302.04761 Local PDF: /Users/moscalej/Documents/Onep/research-papers/2302.04761-toolformer.pdf

Why this paper matters

Toolformer is the paper that made tool use a trainable skill rather than a prompt-engineering trick. Before it, an LM either got tools through large amounts of human-annotated demonstrations or through task-specific few-shot prompts where a human had already decided which tool applies. Toolformer showed that a 6.7B GPT-J can annotate its own pretraining-style corpus with API calls, keep only the calls that demonstrably help it predict future tokens, and finetune on the result — so the model itself learns when to call which tool with what arguments (§1, §2). Landing weeks after ChatGPT and just before the plugin/function-calling era, it is the conceptual ancestor of trained tool-calling in every modern assistant: the idea that tool invocation should be a first-class token-level behavior selected by the model's own utility signal, not by a hand-written router.

The problem

Large LMs hallucinate facts, cannot do precise arithmetic, are blind to the current date, and struggle with low-resource languages — weaknesses where trivially simple external systems (a search engine, a calculator, a calendar) excel (§1). Existing fixes had two failure modes the authors call out explicitly: (i) approaches that rely on large amounts of human annotations of tool use, which are costly and — importantly — encode what humans find useful rather than what the model finds useful; and (ii) approaches that prompt tools in task-specific settings only, so the ability never generalizes beyond the task the prompt was written for (§1). The desiderata: tool use learned self-supervised, without losing generality, with the model deciding for itself when and how to call which tool.

The mechanism

Representing API calls as text (§2)

Every API call is a tuple c = (a_c, i_c) — tool name plus text input. Calls are linearized into the token stream with three special markers:

e(c)    = <API> a_c(i_c) </API>              # call without result
e(c, r) = <API> a_c(i_c) → r </API>          # call with result r

In practice <API>, </API> and → are rendered as [, ] and -> so no vocabulary change is needed (§2, footnote 1). Because calls are just text, the entire method reduces to dataset transformation: take a plain corpus C (a CCNet subset), produce an augmented corpus C* with useful calls spliced in, and finetune on C* with the ordinary LM objective. Since C* contains exactly the same texts as C plus inserted calls, finetuning cannot drift far from the original distribution — this is how generality is preserved (§2, "Model Finetuning").

The five tools (§3)

Constraints on a tool: inputs and outputs representable as text, and a handful of demonstrations available. The paper uses: a question-answering model (Atlas finetuned on Natural Questions), a calculator (four operations, results rounded to two decimals), Wikipedia search (BM25 over the KILT Wikipedia dump), machine translation (600M-parameter NLLB, source language auto-detected with fastText, target always English), and a calendar (returns the current date, takes no input).

Inference (§2)

Decoding proceeds normally until the model emits →. Generation pauses, the harness executes the call, inserts the response followed by </API>, and decoding resumes. This is the interrupt-execute-resume loop every modern function-calling runtime still uses. For evaluation the authors additionally let a call start whenever <API> is among the top k=10 tokens (not only argmax), but allow at most one call per input (§4.2). The k knob directly controls tool eagerness (Table 9, §5):

The algorithm

The heart of the paper: self-supervised annotation of corpus C into C* (§2, Fig. 2).

Input : LM M, corpus C = {x¹,…,x^|C|}, tools A, per-tool prompt P,
        sampling threshold τ_s, filtering threshold τ_f,
        caps k (positions per text) and m (calls per position)
Output: M finetuned on the augmented corpus C*

C* ← ∅
for each text x = x₁…x_n in C:                 # pre-filtered by per-tool
    for each tool a in A:                      #   heuristics (§4.1)
        # 1. SAMPLE candidate call positions
        p_i ← p_M(<API> | P(x), x₁:i−1)   for i = 1…n
        I ← positions with p_i > τ_s, keep top-k
        # 2. SAMPLE candidate calls at each position
        for i in I:
            draw up to m calls c_i¹…c_i^m from M prefixed with
              [P(x), x₁:i−1, <API>]; discard samples lacking </API>
        # 3. EXECUTE every candidate call → response r_i^j
        # 4. FILTER by loss reduction on the continuation
        for each candidate (c_i, r_i):
            keep  iff  L_i⁻ − L_i⁺ ≥ τ_f
    x* ← x with all surviving calls e(c_i, r_i) spliced in at their i
    C* ← C* ∪ {x*}
finetune M on C* with the standard LM objective

The filtering criterion (§2, "Filtering API Calls") is the paper's core idea. Define a weighted cross-entropy of M over the tokens after position i, given some prefix z:

L_i(z) = − Σ_{j=i}^{n}  w_{j−i} · log p_M(x_j | z, x₁:j−1)

L_i⁺ = L_i( e(c_i, r_i) )                     # call AND result as prefix
L_i⁻ = min( L_i(ε), L_i(e(c_i, ε)) )          # best of: no call at all,
                                              #   or call without result
keep c_i  ⇔  L_i⁻ − L_i⁺ ≥ τ_f

Every symbol, in plain language:

• x_j — the tokens of the original text after the candidate call site; the "future" the call is supposed to make easier.
• z — whatever is prepended to the text before scoring: the call with its result, the call without its result, or nothing.
• ε — the empty sequence (no call at all).
• w_t = w̃_t / Σ_s w̃_s with w̃_t = max(0, 1 − 0.2·t) — decaying weights so only roughly the next five tokens matter: an API call must help locally, right where it is inserted (§4.1).
• L_i⁺ — how hard the continuation is when the model sees the call and its answer.
• L_i⁻ — the better of (i) doing nothing and (ii) making the call but never seeing the result. Taking the min makes the test strict: the call survives only if the result, not the call text itself, is what helped.
• τ_f — the margin the loss reduction must clear; per-tool, trades dataset size against precision (Table 2).
• Footnote 3 subtlety: e(c_i, r_i) is scored as a prefix rather than spliced at position i, because the unfinetuned M has never seen mid-text calls and would find them distribution-breaking.

And the outer loop, line by line:

1. Sample positions — "where would a call plausibly start?" answered by the model's own probability of emitting <API> under a few-shot annotation prompt P(x) (Fig. 3); τ_s and top-k keep only confident sites.
2. Sample calls — "what would the call be?" answered by sampling completions from the same model, seeded with <API>; samples that never close with </API> are discarded (footnote 2).
3. Execute — candidates become real: the QA model answers, the calculator computes, BM25 retrieves. Execution "depends entirely on the API itself" — the LM is agnostic to how results are produced (§2, "Executing API Calls").
4. Filter — keep only calls that pay rent in perplexity: L_i⁻ − L_i⁺ ≥ τ_f.
5. Finetune (batch 128, lr 1e-5, linear warmup 10%, §4.1) — distills the filtered self-annotation back into weights, so the model learns to emit calls at exactly the positions, with exactly the arguments, that helped it. Model-in-the-loop end to end; no human labels anywhere.

Yield is small: at τ_f = 1.0 the pipeline keeps ~18.5K QA calls, ~61K Wikipedia-search calls, ~20.6K calendar calls, but only 994 calculator and ~1K translation calls (Table 2); §7 notes that for the calculator, "processing more than a million documents results in only a few thousand examples of useful calls" — the sample-efficiency complaint conceded in the limitations.

Results that matter

All evaluations are zero-shot (no in-context examples), model = GPT-J 6.7B (§4.1–4.2). "GPT-J + CC" is GPT-J finetuned on the same CCNet subset without API calls — the control that isolates tool use from mere finetuning.

The texture behind the table: on LAMA the model chooses the QA tool on 98.1% of examples (§4.2.1); on math it calls the calculator 97.9% of the time and beats models 25× its size (§4.2.2); on QA it relies on Wikipedia search (99.3%) but loses to GPT-3, which the authors attribute to the weak BM25 search and the inability to interact — reformulate queries, browse results (§4.2.3). Perplexity with calls disabled is unchanged vs. the finetuned control (10.3 vs 10.3 on WikiText), so tool training is free in LM quality (Table 8, §4.3). Tool benefit only emerges around ~775M parameters in the GPT-2-family scaling study (§4.4, Fig. 4).

Limitations & how to defend the findings

• "The tools are toys — one-shot, non-interactive calls to a calculator and BM25." True, and §7 concedes it: Toolformer cannot chain tools (calls are sampled independently, so no chained examples exist in C*) nor use a tool interactively (e.g., reformulating a search query). This is the exact gap ReAct-style agent loops later filled. Defense: the contribution is the training signal, not the tool suite; the loss-reduction filter is tool-agnostic.
• "Maybe finetuning on CCNet, not tool use, explains the gains." The paper pre-empts this with the GPT-J + CC control and the "Toolformer (disabled)" ablation: on LAMA and math, disabled Toolformer ≈ baselines while enabled Toolformer more than doubles math performance (Tables 3–4, §4.2.2). The gap is the tools.
• "Is the model actually calibrated about when to call?" Partially. Table 9 (§5) shows that at k=1 decoding the model calls APIs on examples where it would otherwise do worse than average (no-call examples score 44.3 vs 34.9 overall on T-REx) — genuine calibration — but this calibration washes out at k=10, which the paper uses because it maximizes accuracy.
• "Sample efficiency is terrible." Conceded in §7: over a million documents yield only a few thousand useful calculator calls. The suggested fix (iterative/bootstrapped application) is left to future work.
• "The filter could keep spurious calls." Table 10 (§5) shows high L_i⁻ − L_i⁺ usually means intuitively useful calls, but shows counterexamples (a useless WikiSearch scoring 0.92); the authors argue residual noise even acts as regularization against blindly trusting tool output.
• Also conceded (§7): sensitivity to exact wording of the input when deciding to call, and no accounting for the compute cost of a call.

Connections

• Tool-use priors and schemas: the linearization <API> name(input) → result </API> is a minimal tool schema; modern JSON function-calling is the same contract with types. Toolformer is the argument that this contract should be trained in, not just prompted.
• ReAct / agent loops: Toolformer's §7 limitations (no chaining, no interaction) define precisely the space ReAct-style interleaved reason-act trajectories occupy; Toolformer optimizes single-call insertion, agents optimize multi-step episodes.
• Self-supervised data flywheels: the sample-execute-filter-finetune loop is a bootstrapping pattern (the paper cites its kinship to self-training, §6) that reappears in STaR-style reasoning bootstraps and modern synthetic-data pipelines: generate with the model, keep what a measurable criterion validates, train on the survivors.
• Retrieval-augmented LMs (Atlas, REALM): those always prepend retrieved context; Toolformer's filter makes augmentation conditional — retrieve only when it demonstrably lowers loss (§6, "Language Model Pretraining").
• TALM (Parisi et al., 2022) is the closest prior — a similar self-supervised objective but only in task-finetuning settings; Toolformer generalizes it to open corpus annotation (§6, "Tool Use").
• Voyager / skill acquisition: contrast in what is learned — Toolformer bakes tool competence into weights; Voyager stores it as retrievable code. Two poles of the "where does procedural knowledge live" axis.

Reading guide

1. Figure 1 + Figure 2 (pp. 1–2) — the whole method in two pictures; Fig. 2's three-step pipeline is the mental model to retain.
2. §2 Approach — read fully and slowly; the loss definitions around L_i⁺ / L_i⁻ are the paper. Keep footnotes 1–3, they answer the obvious "wait, how?" questions.
3. §3 Tools + Table 1 — skim; just register how simple each tool is.
4. §4.1 Experimental setup — read the weighting function and Table 2 (dataset yield); skim the rest.
5. §4.2 downstream tables (3–7) — read the prose around each table for the tool-usage percentages, which are the real story.
6. §5 Analysis — Table 9 (calibration vs k) and Table 10 (qualitative filter scores) are the two most instructive artifacts; Table 10 is the one table to study — it shows what the objective actually selects.
7. §7 Limitations — read verbatim; it is an unusually honest roadmap of everything the 2023–24 agent literature then built.
8. Appendix A (per-tool prompts and heuristics) only if you plan to reimplement.