ReAct

Full title: ReAct: Synergizing Reasoning and Acting in Language Models Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao (Princeton University; Google Research, Brain team) Venue/year: ICLR 2023 (arXiv v3, March 2023) arXiv: https://arxiv.org/abs/2210.03629 Local PDF: /Users/moscalej/Documents/Onep/research-papers/2210.03629-react.pdf

Why this paper matters

ReAct is the paper that turned "an LLM that talks" into "an LLM that acts," and it did so with nothing but a prompt. Before it, reasoning (chain-of-thought) and acting (action-plan generation for environments) were studied as separate tracks; ReAct interleaves them in one trajectory — Thought → Action → Observation → Thought → … — so that reasoning steers what to do next and environment feedback grounds the next reasoning step (§1, §2). Every modern tool-using agent loop (function calling, browsing agents, coding agents) is structurally a descendant of this pattern. It is also the substrate other papers on this shelf build on: Reflexion uses ReAct as its actor, and Tree of Thoughts generalizes the "thought" half into explicit search.

The problem

Two failure modes existed on either side of the gap (§1):

• Reasoning without acting. Chain-of-thought (CoT) is a "static black box": the model reasons only over its own internal representations, is not grounded in the external world, and cannot update its knowledge — leading to fact hallucination and error propagation across the chain (§1, Figure 1(1b)). The paper's human study quantifies this: hallucination accounts for 56% of CoT's failure cases and gives CoT a 14% false-positive rate even among its "successes" (Table 2, §3.3).
• Acting without reasoning. Action-only agents (imitation/RL policies, or WebGPT-style action generation) never verbalize high-level goals, plans, or state tracking, so they cannot decompose tasks, handle exceptions, or notice when a plan is stale. In the paper's controlled "Act" ablation, the agent "fails to correctly decompose goals into smaller subgoals, or loses track of the current state of the environment" (§4, Results), and gets stuck repeating actions (Figure 1(2a)).

The mechanism

Augmenting the action space with language (§2)

ReAct starts from the standard agent formalism: at step t the agent sees observation o_t, holds context c_t = (o_1, a_1, ..., o_{t-1}, a_{t-1}, o_t), and picks action a_t via a policy pi(a_t | c_t). The one-line idea:

 = A ∪ L        # augment the environment's action space A with the language space L

An action â_t ∈ L is a thought: it does not touch the environment and returns no observation; its only effect is to be appended to the context, c_{t+1} = (c_t, â_t), where it can support future reasoning or acting (§2). Thoughts serve identifiable roles: decomposing goals and creating plans, injecting commonsense knowledge, extracting salient parts of observations, tracking progress, and handling exceptions by revising the plan (§2, referencing Figure 1).

Because L is unlimited, learning this augmented policy from scratch would be hopeless; ReAct instead relies on a frozen large LM (PaLM-540B) prompted with few-shot in-context trajectories — human-written examples of interleaved thoughts, actions, and observations (§2). Density is a design dial: for knowledge tasks, thoughts and actions strictly alternate; for long-horizon decision-making tasks, thoughts appear sparsely, only at key decision points, and the model decides when to think (§2, §4 ALFWorld).

Knowledge-intensive tasks: HotpotQA and FEVER (§3)

The environment is a deliberately minimal Wikipedia API with three actions (§3.1): search[entity] (first 5 sentences of the entity's page, or top-5 similar entity suggestions), lookup[string] (next sentence containing the string — simulated Ctrl-F), and finish[answer]. The setup is question-only: no gold paragraphs are given, so the model must retrieve everything through explicit reasoning-driven search. Prompts use 6 (HotpotQA) and 3 (FEVER) hand-written trajectories (§3.2); more exemplars did not help (footnote 2).

Baselines are exact ablations of the ReAct trajectories (§3.2): Standard (remove thoughts, actions, observations), CoT (remove actions/observations; reason only), CoT-SC (self-consistency over 21 sampled CoT chains at temperature 0.7), and Act (remove thoughts).

Two hybrid strategies combine internal and external knowledge (§3.2): ReAct → CoT-SC (run ReAct; if it fails to answer within 7 steps on HotpotQA / 5 on FEVER, back off to CoT-SC) and CoT-SC → ReAct (if the CoT-SC majority answer wins fewer than half the n samples — low internal confidence — back off to ReAct). These hybrids are the best prompting methods overall (Table 1), and reach 21-sample CoT-SC performance with only 3–5 samples (Figure 2).

Decision-making tasks: ALFWorld and WebShop (§4)

On ALFWorld (text household games, 134 unseen eval games, >50 steps horizon), ReAct prompts use just 2 annotated trajectories per task type with sparse thoughts that decompose goals, track subgoals, and reason with commonsense about where objects are likely to be found (§4). On WebShop (1.18M real products, 12k instructions, 500 test instructions), ReAct adds reasoning about what to explore, when to buy, and which options match the instruction (§4). Both beat imitation and imitation+RL baselines trained on thousands of trajectories, with one or two in-context examples (abstract, §4).

A key controlled comparison: ReAct-IM, an Inner-Monologue-style ablation whose "thoughts" are limited to observing the environment state and stating what remains to be done. It scores 53 vs ReAct's 71 on ALFWorld (Table 3, §4) — evidence that flexible, sparse reasoning (goal decomposition, commonsense) is what carries the value, not just any interleaved feedback text.

Fine-tuning (§3.3, Figure 3)

Bootstrapping 3,000 correct ReAct trajectories and fine-tuning smaller PaLM models flips the ranking: prompted ReAct is the worst of four methods on PaLM-8/62B (interleaved format is hard to learn in-context for small models), but fine-tuned ReAct on PaLM-8B beats all PaLM-62B prompting methods, and fine-tuned PaLM-62B ReAct beats all 540B prompting methods. Fine-tuning Standard/CoT instead teaches the model to memorize (potentially hallucinated) facts; fine-tuning ReAct teaches a generalizable skill: how to reason and act to access knowledge (§3.3).

The algorithm

The paper never isolates pseudocode, but §2–§3 fully determine the loop:

# ReAct(question, few_shot_prompt, env, max_steps)
c ← few_shot_prompt + question              # context = exemplar trajectories + task
for t = 1 .. max_steps:
    â ← LM(c)                               # greedy decode next segment
    if â is a Thought:                      #   â ∈ L: free-form reasoning text
        c ← c + â                           #   no env call, no observation
    else:                                   #   â ∈ A: e.g. search[x], lookup[s], finish[y]
        o ← env.execute(â)                  #   act on the environment
        c ← c + â + o                       #   append action AND observation
        if â == finish[y]: return y
return failure                              # optionally: back off to CoT-SC (§3.2)

Walkthrough:

1. Context assembly. The prompt is a handful of full human trajectories (thought/action/ observation interleaved) followed by the new task. This is the entire "training" signal.
2. One decode per step. The frozen LM greedily generates the next segment. Nothing in the architecture forces a Thought vs an Action; the exemplars establish the format, and for decision-making tasks the LM chooses asynchronously when a thought is worth emitting (§2).
3. The branch is the whole trick. A Thought only mutates the context — it is a self-issued note that shapes the distribution of the next decode. An Action leaves the language space, hits the environment, and imports an observation the model could never have generated itself. That imported text is what kills hallucination (Table 2: 0% hallucination among ReAct failures vs 56% for CoT).
4. Termination. finish[answer] ends the episode; step budgets (7 / 5 on HotpotQA / FEVER) bound the loop, and exceeding them can trigger the CoT-SC fallback (§3.2).

There are no equations beyond the policy notation above; ReAct's contribution is the action-space union and the prompting format, not new math.

Results that matter

The abstract's headline: ReAct beats imitation/RL methods on ALFWorld and WebShop "by an absolute success rate of 34% and 10% respectively," from one- or two-shot prompts.

Limitations & how to defend the findings

• "ReAct loses to CoT-SC on HotpotQA (27.4 vs 33.4)." True, and the paper says so plainly (§3.3). The defense is diagnostic, not defensive: Table 2 shows ReAct trades hallucination (0% vs 56%) for reasoning inflexibility — the rigid thought-action-observation structure raises reasoning-error rate (47% vs 16%), and 23% of failures are non-informative search results derailing the chain. That is precisely why the hybrid ReAct↔CoT-SC methods are the paper's actual best systems (35.1 / 64.6).
• "Is it the reasoning, or just having tools?" The Act ablation isolates this: identical environment, identical exemplars minus thoughts. ReAct beats Act everywhere (Table 1, 3, 4); on ALFWorld the advantage is consistent across all six prompt permutations, averaging 62% relative gain (§4). ReAct-IM further shows what kind of reasoning matters (71 vs 53, §4).
• "A repetitive-loop failure mode exists." Acknowledged: the model sometimes re-generates prior thoughts/actions and cannot escape; the authors classify it under reasoning error and suspect greedy decoding, suggesting beam search as future work (§3.3, footnote 4).
• "Prompting-based results are far from SoTA." The paper agrees (Table 1 vs supervised 67.5/89.5) and offers fine-tuning as the path: with only 3,000 bootstrapped trajectories, fine-tuned ReAct dominates at every model scale tried (§3.3, Figure 3).
• "Only one base model." Main results are PaLM-540B, but Appendix A.1 reports GPT-3 experiments that outperform PaLM-540B (footnote 1, Reproducibility Statement), supporting generality across LMs.
• Scaling caveat. Complex tasks with large action spaces need more demonstrations than in-context limits allow — stated in the conclusion (§6) as the motivation for fine-tuning.

Connections

• The agent loop. ReAct is the minimal viable agent loop: LLM decode → tool call → observation → decode. Modern function-calling APIs institutionalize the same  = A ∪ L split.
• Self-consistency. CoT-SC is both a baseline and a component: the ReAct↔CoT-SC hybrids (§3.2) are an early "route between internal and external knowledge" policy — see self-consistency.md for the voting math being invoked.
• Reflexion. Uses ReAct as its Actor and adds an episodic memory of verbal self-reflections across trials — directly attacking ReAct's repetitive-loop and hallucination-persistence failure modes (see reflexion.md).
• Tree of Thoughts. ToT takes the "thought" half and makes it a search tree with lookahead and backtracking; ReAct's single greedy trajectory is the degenerate depth-first path (see tree-of-thoughts.md).
• Inner Monologue / SayCan. The closest priors (§5): IM injects environment feedback as text but has no flexible reasoning — the ReAct-IM ablation is the head-to-head.
• Interpretability. The trajectory is the explanation: humans can distinguish model-internal knowledge from environment facts and even edit thoughts mid-run to steer the agent (§2 D, Figure 5 referenced in §4).

Reading guide

• Read first: §1 (framing), Figure 1 (the four-method comparison — the whole paper in one figure), §2 (the formalism; two pages, all the ideas).
• Then: §3.2–3.3 with Table 1 and Table 2 — the ablation logic and the honest failure analysis are the most instructive parts.
• Then: §4 Results paragraphs + Table 3/4, especially the ReAct-IM comparison.
• Skim: §5 Related Work; Appendix C (full prompts — worth one pass to see exactly how sparse the supervision is); Appendix D (trajectory examples), A.1 (GPT-3 replication).
• The one table to study: Table 2 (success/failure mode taxonomy). It explains why grounding beats pure reasoning, why ReAct still fails, and why the hybrids win — the causal story behind every number in Table 1.