Voyager

Full title: VOYAGER: An Open-Ended Embodied Agent with Large Language Models Authors: Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi "Jim" Fan, Anima Anandkumar (NVIDIA, Caltech, UT Austin, Stanford, UW Madison) Venue/year: arXiv, May 2023 (v2 Oct 2023, cs.AI); project site https://voyager.minedojo.org arXiv: https://arxiv.org/abs/2305.16291 Local PDF: /Users/moscalej/Documents/Onep/research-papers/2305.16291-voyager.pdf

Why this paper matters

Voyager is "the first LLM-powered embodied lifelong learning agent" (abstract): an agent that plays open-ended Minecraft indefinitely, sets its own goals, writes executable code to achieve them, and keeps what it learns as an ever-growing library of skills. Its historical importance is the triad it introduced — automatic curriculum + skill library + iterative prompting with environment feedback — which became the canonical architecture for LLM agents that must improve over long horizons without gradient updates. Everything happens through black-box GPT-4 queries: no finetuning, no RL, no parameter access (§1). Skills are stored as interpretable, composable programs retrieved by embedding similarity, which both compounds ability over time and sidesteps the catastrophic forgetting that plagues continual learning (§1, §2.2). If Toolformer is "tools in the weights," Voyager is "skills in an external memory" — the founding example of procedural memory for LLM agents.

The problem

LLM agents circa early 2023 (ReAct, Reflexion, AutoGPT) could execute a given task by reasoning in context, but they were not lifelong learners: nothing accumulated between tasks, so complex behaviors had to be re-derived from scratch every time, and long-horizon domains stalled them completely (§1). Classical alternatives — RL and imitation learning on primitive actions — struggle with systematic exploration, interpretability, and generalization in open worlds (§1). Minecraft is the stress test: no fixed storyline, a deep tech tree (wood → stone → iron → diamond), and procedurally generated terrain, so an effective agent must (1) propose tasks matched to its current competence, (2) refine skills from feedback, (3) commit mastered skills to memory for reuse, and (4) keep exploring in a self-driven way (§1). No prior LLM technique did all four.

The mechanism

Voyager is three modules wired around a Minecraft simulation (MineDojo + the Mineflayer JavaScript API, §3.1); code — not low-level motor commands — is the action space, because "programs can naturally represent temporally extended and compositional actions" (§1).

Automatic curriculum (§2.1)

GPT-4 is prompted to propose the next task from four input components (§2.1):

1. Directives encoding the overarching goal — "My ultimate goal is to discover as many diverse things as possible ... The next task should not be too hard since I may not have the necessary resources or have learned enough skills to complete it yet."
2. The agent's current state — inventory, equipment, nearby blocks and entities, biome, time, health and hunger bars, position.
3. Previously completed and failed tasks — the agent's exploration frontier.
4. Additional context — GPT-3.5 self-asks and answers questions about the situation.

The result is a bottom-up curriculum of "increasingly complex" tasks paced to the agent's frontier — an in-context form of novelty search (§2.1, Fig. 3).

Skill library (§2.2)

Each mastered behavior is stored as an executable program. Indexing: GPT-3.5 writes a docstring-style description of the program; its embedding (text-embedding-ada-002) is the key, the code is the value (Fig. 4 top). Retrieval: for a new task, GPT-3.5 first drafts a general plan for solving it; the embedding of that plan plus current environment feedback is the query, and the top-5 skills are returned into the code-generation prompt (Fig. 4 bottom). Complex skills are synthesized by composing simpler retrieved ones, which "compounds Voyager's capabilities rapidly over time and alleviates catastrophic forgetting" (§1, §2.2).

The code-generation prompt to GPT-4 carries five components (§2.2):

1. Guidelines — e.g., "Your function will be reused for building more complex functions. Therefore, you should make it generic and reusable."
2. Control primitive APIs and retrieved skills — the in-context library the new program may call.
3. Last round's code, environment feedback, execution errors, and critique — the debugging context.
4. The agent's current state — same fields as the curriculum sees.
5. Chain-of-thought prompting — reason before code.

Iterative prompting mechanism (§2.3)

LLMs rarely emit correct embodied code in one shot, so each task is a refinement loop over three feedback channels: (1) environment feedback — intermediate execution progress surfaced via bot.chat() (e.g., "I cannot make an iron chestplate because I need: 7 more iron ingots"); (2) execution errors — interpreter tracebacks; (3) self-verification — a separate GPT-4 agent is shown the agent's state and the task, acts as a critic, judges success, and on failure emits a critique suggesting how to finish (Fig. 6). The authors note self-verification is more comprehensive than Reflexion-style self-reflection because it both checks success and diagnoses failure (§2.3).

The algorithm

Reconstructed from §2 and Fig. 2:

skill_library ← ∅ ;  completed ← [] ;  failed ← []
state ← observe(world)
loop forever:                                  # lifelong exploration
    # --- 1. AUTOMATIC CURRICULUM proposes the next task (§2.1)
    qa ← GPT35.self_ask_and_answer(state, progress)
    task ← GPT4.curriculum(directives, state, completed, failed, qa)

    # --- 2. ITERATIVE PROMPTING synthesizes code for the task (§2.3)
    code, env_fb, err, critique ← ∅
    for round = 1 … 4:                         # give up after 4 rounds
        plan   ← GPT35.general_suggestion(task)
        skills ← skill_library.top5( embed(plan + env_fb) )   # §2.2
        code   ← GPT4.codegen( guidelines,
                               primitives + skills,           # in-context
                               (code, env_fb, err, critique), # last round
                               state, chain_of_thought )
        env_fb, err, state ← execute(code)     # Mineflayer / MineDojo
        success, critique ← GPT4.self_verify(state, task)
        if success: break

    # --- 3. UPDATE memory and exploration progress (§2.2)
    if success:
        desc ← GPT35.describe(code)
        skill_library[ embed(desc) ] ← code    # add new skill
        completed.append(task)
    else:
        failed.append(task)                    # curriculum may retry later

Line by line:

• loop forever — the outer loop never terminates: there is no final goal, only the standing directive to maximize diverse discoveries. Lifelong learning is the loop structure itself.
• qa ← GPT35.self_ask_and_answer(...) — cheap context enrichment: GPT-3.5 asks and answers questions about the current situation ("What can I craft with 4 raw iron and a furnace?"); GPT-3.5 rather than GPT-4 "due to budgetary considerations" (§2.1).
• task ← GPT4.curriculum(...) — the curriculum call. GPT-4 sees what the agent has and has not done and proposes one feasibility-matched task — e.g., "Craft 1 stone pickaxe" when the inventory holds a wooden pickaxe and three stones (Fig. 3). Failed tasks in the prompt keep it from re-proposing something still out of reach.
• for round = 1…4 — bounded persistence: after 4 failed code-generation rounds, control returns to the curriculum, which proposes something else. Failure is logged, not fatal — the task can be re-proposed later when the agent is stronger (§2.3, §4 "Inaccuracies").
• plan ← GPT35.general_suggestion(task) — retrieval queries are built from a plan, not the raw task name, so "craft iron pickaxe" retrieves smelting and mining skills, not just string-similar ones (Fig. 4 bottom).
• skills ← top5_retrieve(...) — up to five mastered programs enter the prompt as in-context libraries the new code can call; this is how skills compose (§2.2).
• code ← GPT4.codegen(...) — the prompt carries the previous round's code, environment feedback, execution errors, and the critic's critique, so each iteration is a debugging step, not an independent resample (§2.2, Fig. 5).
• execute(code) — the program runs in Minecraft through Mineflayer; bot.chat() calls inside control primitives surface intermediate progress as environment feedback (§2.3).
• GPT4.self_verify(state, task) — a second GPT-4 instance acts as the reward model of the loop: given only the resulting state and the task, it declares success or emits a critique ("You need to kill one more sheep", Fig. 6).
• memory write — only verified programs enter the library, keyed by the embedding of a GPT-3.5-written description. The gate is what keeps the library trustworthy enough to compose from.
• Determinism — all temperatures 0 except the curriculum's 0.1, the loop's only stochasticity: task diversity (§3.1). Models: gpt-4-0314, gpt-3.5-turbo-0301, text-embedding-ada-002 (§3.1).

Results that matter

Baselines (ReAct, Reflexion, AutoGPT) are re-implemented on the same MineDojo/Mineflayer stack with the same GPT-4 and the same observations (§3.2). Numbers are prompting iterations, averaged over three trials; fractions = successful trials.

ReAct and Reflexion unlock no tech-tree level within the 160-iteration cap (Table 1) — with the same GPT-4 underneath. | Map distance traversed | 1× | 2.3× longer | Fig. 7, §3.3 | | Unseen tasks (Diamond Pickaxe / Golden Sword / Lava Bucket / Compass), fresh world | AutoGPT: 0/4 solved; +Voyager's library: solves 3/4 | all 4 solved, e.g. 19±3, 18±7, 21±5, 18±2 iters (3/3) | Table 2, Fig. 8 |

Ablations (§3.4, Fig. 9) — each maps a component to a measured cost:

Notably, bolting Voyager's learned skill library onto AutoGPT also improves AutoGPT (Table 2) — the library is a portable asset, not an entangled component. And with human feedback substituting for its critic and curriculum, Voyager builds complex 3D structures like a Nether portal and a house (§3.5, Fig. 10).

Limitations & how to defend the findings

• "It's not learning — GPT-4 already knows Minecraft from pretraining." Partly fair: the curriculum explicitly "capitalizes on the internet-scale knowledge contained within GPT-4" (§2.1). But the ablations show pretrained knowledge alone is insufficient — the same GPT-4 inside ReAct/Reflexion unlocks zero tech-tree levels (Table 1), and random-curriculum GPT-4 loses 93% of discoveries. The delta is the architecture, not the model.
• "Prompting iterations are a weird x-axis." True, but it is a fair cost unit since all methods share the same LM and simulator; Table 1 counts iterations to milestone with a 160-iteration cap, so it doubles as a success-rate measure.
• "The tech tree could be memorized; does anything generalize?" The zero-shot test (§3.3) clears inventory and instantiates a new world; Voyager solves all four unseen tasks 3/3 while baselines solve none within 50 iterations (Table 2). The skill library even transfers to a different agent (AutoGPT).
• "Self-verification is an LLM grading an LLM." The authors concede it occasionally fails (e.g., not recognizing spider string as a sign of beating a spider, §4 "Inaccuracies"). Its defense is empirical: removing it is the single most damaging ablation (−73%), so whatever noise it carries, the signal dominates.
• "Hallucinations." Conceded in §4: the curriculum sometimes proposes impossible items ("copper sword"); codegen sometimes invents functions or uses cobblestone as fuel. The iterative loop absorbs much of this as recoverable error.
• "Cost and privileged access." GPT-4 API is "15× more expensive than GPT-3.5" (§4), and Voyager reads ground-truth state via Mineflayer rather than pixels — the paper explicitly scopes out the perception problem and declines apples-to-oranges comparison with pixel-based agents like VPT (§3.2).

Connections

• Procedural memory / skill libraries: Voyager is the reference implementation of agent procedural memory — verified code keyed by embedded descriptions — as distinct from episodic stores (MemGPT, Generative Agents' memory stream). The write-gate ("only verified skills enter") is the design point most successors kept.
• Toolformer: two answers to "where do capabilities accumulate?" — Toolformer distills tool use into weights via a loss-based filter; Voyager accumulates it in retrievable code via an LLM critic. Voyager's no-gradient stance is what makes it work on a closed API model.
• Reflexion: Voyager's self-verification generalizes Reflexion's self-reflection — it not only reflects on failure but actively decides task success, i.e., it is the reward model of the loop (§2.3).
• Curriculum & novelty search: the automatic curriculum is in-context intrinsic motivation ("discover as many diverse things as possible"), an LLM stand-in for the curiosity objectives of the RL exploration literature (§2.1).
• Code-as-action agents: choosing executable programs over primitive actions (§1) anticipates code-first agent frameworks; the environment-feedback-into-prompt loop is the same shape as a coding agent iterating against a test harness — SWE-agent's edit/run/lint loop is this triad transplanted to repositories.
• ACI framing: Mineflayer's high-level JS API plays the role SWE-agent later names the "agent-computer interface": a hand-built action layer that makes the environment legible to an LM.

Reading guide

1. Figure 2 (p. 3) — the whole system on one page; keep it beside you for §2.
2. §2.1–2.3 Method — the core read; Figs. 3 (curriculum prompts), 4 (skill add/retrieve), 5 (feedback types), 6 (self-verification) each carry a mechanism the prose only sketches. Figure 4 is the one figure to study — both the write path and the read path of the skill library live in it.
3. §3.1–3.2 — skim, but note the baseline re-implementations and the explicit non-comparison to pixel-based agents.
4. §3.3 + Tables 1–2 + Fig. 1 — the headline results; check the fractions (x/3) as much as the means.
5. §3.4 + Fig. 9 — the ablations are the paper's real argument; read fully.
6. §4 Limitations — short and candid; read verbatim.
7. Appendix A (full prompt texts) if you want to reimplement — the prompts are the implementation.