MemGPT

Full title: MemGPT: Towards LLMs as Operating Systems Authors: Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez (UC Berkeley) Venue/year: arXiv preprint, 2023 (v2: Feb 2024) — arXiv:2310.08560 Links: arXiv · local PDF: research-papers/2310.08560-memgpt.pdf

Why this paper matters

MemGPT is the paper that turned "the context window is too small" from a model-scaling problem into a systems problem. Instead of waiting for longer contexts, it treats the LLM's fixed context window as physical RAM and builds an OS-style virtual memory layer on top: the model itself pages data between in-context "main memory" and out-of-context disk-like stores using function calls (§1). This one move — the LLM manages its own memory via tools, driven by interrupts and warnings — became the template for a generation of stateful agent frameworks (Letta, the direct descendant, plus the memory-block designs copied across the industry). Arriving in late 2023, it sits between Generative Agents' memory stream (retrieval-only, no self-editing) and today's production agent memory systems, and it is the first to make eviction, summarization, and retrieval agent-initiated rather than pipeline-hardcoded.

The problem

Fixed context windows cap everything an agent can be. In early 2024 the widely deployed models spanned 2k–200k tokens (Table 1), which in chat terms is a few dozen to a few thousand messages — nowhere near "a companion that remembers weeks of conversation" or "analyze an SEC 10-K that passes a million tokens" (§3.2). Naively extending context is expensive (attention is quadratic, §1) and doesn't even work well: the paper leans on the "lost in the middle" result (Liu et al., 2023a) showing long-context models attend unevenly, recalling the beginning and end of the window far better than the middle (§3.2). Retrieval-augmented pipelines existed, but the pipeline decided what got retrieved and when; the model couldn't notice its own context filling up, decide what was worth keeping, or iterate on a failed search. What was missing was self-directed memory management under explicit awareness of token limits.

The mechanism

Two memory tiers (§2)

• Main context — the prompt tokens; anything here is visible to the LLM at inference time. Analogous to RAM.
• External context — anything outside the window; must be explicitly moved into main context via function calls to be seen. Analogous to disk. Two databases back it: recall storage (the full message history, searchable) and archival storage (a read/write database of arbitrary-length text objects, e.g. uploaded documents; vector search via pgvector in the experiments, §3.2.1).

Main context is split into three sections (§2.1)

1. System instructions — read-only, static. Describe the memory hierarchy and the function schema the LLM can call.
2. Working context — a fixed-size read/write block of unstructured text, writeable only via MemGPT functions. For chat, it holds key facts about the user and the agent's persona (this is the ancestor of Letta's "memory blocks").
3. FIFO queue — the rolling message history (user/agent messages, system warnings, function call I/O). Its first slot always holds a system message containing a recursive summary of everything evicted so far.

The queue manager (§2.2)

The queue manager is the "kernel" component. It appends incoming messages to the FIFO queue, triggers LLM inference, and writes both incoming messages and LLM outputs to recall storage. It also enforces the eviction policy — see the algorithm below. Crucially, evicted messages are never lost: they leave the window but remain in recall storage, retrievable by search.

The function executor and self-directed editing (§2.3)

All memory movement is via LLM-emitted function calls: append/replace working context, search recall storage, read/write archival storage. The system prompt tells the model how and when to use them, and MemGPT feeds back function results — including runtime errors like "working context is at maximum capacity" — so the model can correct itself. Retrieval is paginated so a search result can never itself blow the context budget.

Events, interrupts, and heartbeats (§2.4)

LLM inference is triggered by events: user messages, system messages (e.g. the memory-pressure warning), user interactions (login, document-upload alerts), and timed events that let the agent run "unprompted." Function calls can carry a request_heartbeat=true flag: if set, control returns immediately to the LLM after the function executes, letting it chain calls (e.g. page through search results across multiple steps); if absent, the processor yields until the next external event. Function chaining is what makes multi-hop retrieval possible (Figure 3, Figure 8).

The algorithm

Reconstructed from §2.2 (queue manager, eviction, recursive summary) and §2.4 (events, heartbeats):

# Persistent state
main_context   = [system_instructions, working_context, fifo_queue]
recall_storage = message DB          # everything ever seen
archival_store = document DB         # arbitrary text objects
fifo_queue[0]  = recursive_summary   # of all evicted messages

loop forever:
    event = wait_for_event()                     # user msg | system alert | timer | login
    fifo_queue.append(parse_to_text(event))
    recall_storage.write(event)

    # --- memory pressure check (queue manager) ---
    if token_count(main_context) > WARNING_FRACTION * window:      # e.g. 0.70
        fifo_queue.append(system_msg("memory pressure warning"))
    if token_count(main_context) > FLUSH_FRACTION * window:        # e.g. 1.00
        evicted = fifo_queue.pop_oldest(EVICT_FRACTION * window)   # e.g. 0.50
        recursive_summary = summarize(recursive_summary, evicted)
        fifo_queue[0] = recursive_summary
        # evicted messages persist in recall_storage

    # --- inference + function chaining ---
    heartbeat = true
    while heartbeat:
        output = LLM(concat(main_context))
        recall_storage.write(output)
        call = parse_and_validate(output)        # parse errors fed back to LLM
        result = execute(call)                   # may error, e.g. "context full"
        fifo_queue.append(result)
        heartbeat = call.request_heartbeat       # absent flag = yield

Walkthrough, line by line:

• wait_for_event / append / write — everything that happens is both shown to the model (queue) and journaled (recall storage). The journal is what makes eviction safe.
• Warning threshold (~70%) — the queue manager injects a "memory pressure" system message. This is an interrupt, not an action: it gives the LLM a window of turns to rescue important facts from the queue into working context or archival storage before they get evicted (Figure 1 shows exactly this: the agent appends "Birthday is February 7" to working context after the alert).
• Flush threshold (~100%) — the queue manager acts unilaterally: it evicts a fixed chunk of the oldest messages (e.g. half the window's worth) and folds them into a new recursive summary computed from the old summary plus the evicted messages. Recursion is what keeps the summary bounded no matter how long the conversation runs.
• Recursive summary at index 0 — the model always sees a lossy digest of deep history for free, and can call recall_storage.search(...) for the lossless version (Figure 2's "six flags" lookup).
• Parse/validate + error feedback — MemGPT validates the emitted function call before executing; both parse errors and runtime errors are appended to context, forming the learn-from-feedback loop of §2.3.
• Heartbeat loop — request_heartbeat=true re-invokes the LLM immediately with the function result in context; the model iterates (page 2 of search results, then page 3, then answer). No flag means yield: the turn ends until the next event.

Results that matter

Notes on the setups: DMR asks a question about sessions 1–5 of a Multi-Session Chat conversation, judged by GPT-4 plus ROUGE-L recall (§3.1.1). Baselines get a lossy summary of past sessions; MemGPT gets paginated search over the full history. The nested KV task fixes 140 UUID pairs (~8k tokens) and varies nesting 0–4 over 30 orderings (§3.2.2).

Limitations & how to defend the findings

• "The baselines are handicapped — MemGPT gets the full history, baselines get a summary." True, but that is the claim: with a fixed window you must compress lossily, and the paper shows the compression is what kills accuracy (§3.1.1). The fair comparison is capability-per-fixed-window, and the truncation study in Figure 5 makes the same point for documents: shrink documents to fit and accuracy falls as the gold snippet's omission chance grows.
• "It depends entirely on frontier function calling." The paper concedes this openly: MemGPT with GPT-3.5 is "significantly degraded... due to its limited function calling capabilities, and performs best using GPT-4" (§3.2.1), and on nested KV even MemGPT+GPT-4 Turbo underperforms MemGPT+GPT-4 (Figure 7). The mechanism is sound but its floor is the base model's tool-use reliability — a real deployment constraint in 2023, much less so now.
• "Retrieval is still bottlenecked by embedding search." The paper agrees: gold documents "often appear outside of the first dozen retrieved results, or not even further" (§3.2.1). MemGPT's answer is iterative paging — it can keep calling the retriever — but it also "will often stop paging through retriever results before exhausting the retriever database" (§3.2.1). Self-directed retrieval mitigates, not solves, retriever noise.
• "The thresholds (70%/50%) are magic numbers." They are illustrative settings, not tuned constants — the paper presents them with "e.g." (§2.2) and never ablates them. There is no study of summary drift over very long horizons either; recursive summarization can accumulate error and nothing in the eval spans enough sessions to measure it.
• "LLM-as-judge evaluation." DMR and doc-QA correctness use a GPT-4 judge (§3.1.1, Appendix 6.1.2/6.1.5); the paper cites judge–human agreement work (Zheng et al., 2023) and pairs the judge with ROUGE-L(R) to hedge verbosity effects.

Connections

• MemGPT → Letta. The research codebase became Letta; working context became named, individually editable memory blocks, and the heartbeat/event loop became the agent runtime. When our sandbox's M3 milestone skips eviction, it is skipping exactly the §2.2 queue-flush machinery — keeping self-editing working context (Figure 1/4) while letting the FIFO queue grow, a legitimate simplification when windows are 200k+ tokens.
• CoALA's taxonomy locates MemGPT precisely: working context ≈ working memory, recall storage ≈ episodic memory, archival storage ≈ semantic memory, and the function schema is procedural memory. MemGPT is a concrete instance of CoALA's "internal memory actions."
• Generative Agents solved remembering by retrieval scoring over an append-only stream; MemGPT solves it by self-editing plus paging. The two are complementary: one asks "what should I fetch," the other "what should I keep resident."
• Nested KV (§3.2.2) is proto-multi-hop-RAG: the same collate-across-sources problem that AriGraph, HippoRAG, and Zep later attack with graph structure, MemGPT attacks with sequential re-querying via heartbeats.
• The OS metaphor generalizes: memory pressure interrupts ≈ page-fault handling, recursive summary ≈ swap compression, heartbeats ≈ kernel re-entry. The paper's closing pitch — "bridging concepts from OS architecture into AI systems" (§5) — is the same design stance behind context-management features in modern agent harnesses (compaction, context editing).

Reading guide

• Read first: §2 end-to-end (2.1 main context → 2.2 queue manager → 2.3 function executor → 2.4 control flow). It is only ~2 pages and contains the entire system.
• Then: §3.1.1 (DMR) + Table 2 for the headline conversational result; §3.2.2 + Figure 7 for the multi-hop story.
• The one figure to study: Figure 3 — the full architecture with prompt-token layout, queue manager, function executor, and both storage tiers; every arrow corresponds to a sentence in §2.
• Skimmable: §3.1.2 (opener task — a soft engagement metric), §4 related work, Table 1 (context-length snapshot, now dated).
• Appendix worth a look: §6.1.1/6.1.6 — the actual system-prompt personas ("DO NOT STOP SEARCHING UNTIL YOU VERIFY THAT THE VALUE IS NOT A KEY") show how much of the behavior is prompt-engineered, useful calibration for reproducing the results.