Memory & Context

An agent has no memory of its own. A language model is a stateless function: each call maps a sequence of input tokens to output tokens, and once the call returns, nothing persists. Every "memory" an agent appears to have is something a developer put back into the prompt on the next turn. Understanding agent memory therefore means understanding two distinct things and the machinery that moves information between them: the context-window — the finite slice of tokens the model can attend to right now — and durable memory — everything stored outside that window and selectively reintroduced.

This distinction is the heart of the matter. The context window is working memory; it is volatile and small. Long-term memory is storage; it is durable and effectively unbounded, but the model cannot see it until something retrieves the relevant piece and writes it into the window. The job of an agent's memory system is to decide, on every turn, which fraction of everything-ever-known deserves a slot in that scarce window.

The context window is a finite resource

The context window is the maximum number of tokens — prompt plus generation — a model can process in a single forward pass. Modern windows are large (hundreds of thousands of tokens), but large is not the same as free. Anthropic frames the window as a resource that "must be treated as a finite resource with diminishing marginal returns," and points to context rot: the phenomenon — measured on needle-in-a-haystack benchmarks — where, as the number of tokens in the window grows, the model's ability to accurately recall any specific piece of that context decreases. Anthropic situates this alongside two architectural tensions that work against long contexts: a transformer forms n² pairwise relationships for n tokens, and models are trained on comparatively little long-sequence data. The article presents these as related pressures rather than a single proven cause of context rot, but the practical upshot is the same — more tokens are not automatically more useful.

The practical consequence is that you cannot solve memory by simply pasting everything into the prompt. A window full of stale or off-topic content makes the model worse, not better. This is why memory is an active discipline of selection rather than passive accumulation — the subject of context-engineering-managing-a-scarce-window">context engineering below.

Short-term memory: conversation history and scratchpads

Short-term-memory is the information an agent retains within a single conversation or session. Concretely it is the running transcript — the alternating sequence of user messages, model responses, tool calls, and tool results — that grows turn over turn and is replayed into the context window at the start of each new turn. LangChain describes it as letting an application "remember previous interactions within a single thread or conversation," persisted between turns by a checkpointer that saves state to a database (or in-memory) and keyed by a thread (analogous to how email groups messages into one conversation).

Short-term memory also includes the agent's scratchpad: intermediate reasoning, plans, and working notes the model writes for itself and reads back on later steps. In the agent loop, the scratchpad is what carries a multi-step plan across iterations without it having to be re-derived each time.

Because the transcript grows without bound while the window does not, every agent framework needs a strategy for when history exceeds the window. LangChain documents three concrete approaches:

• Trimming — remove the first or last N messages before calling the model (e.g. a sliding window over recent turns).
• Deleting — permanently drop specific messages from the persisted state.
• Summarizing — replace a span of older messages with a model-generated summary, trading fidelity for tokens.

Trimming and deletion are cheap but lossy in an all-or-nothing way; summarization preserves the gist of more history at the cost of detail and an extra model call.

Long-term memory: storage that outlives the window

Long-term-memory is information that persists across conversations, sessions, and threads — user preferences learned weeks ago, facts about a domain, examples of past successful behavior. It lives in external storage, not in the transcript, and is pulled into the window only when relevant. LangChain implements this as a store that saves data as JSON documents organized by a namespace (a tuple that acts like a folder path, e.g. ("users",) or (user_id, "memories")) and a separate key that names the document within that namespace (e.g. "user_123"). A memory is written with store.put(namespace, key, value) — for example store.put(("users",), "user_123", value) — and read back either by direct lookup (store.get) or by store.search, which supports both metadata filters and — when an embedding index is configured — semantic vector similarity.

Cognitive science gives the field a useful taxonomy, adopted by LangMem and the LangGraph memory docs, for what kind of thing is being remembered:

• Semantic-memory — facts and general knowledge that ground responses ("the user's company uses Postgres"). Often stored either as an unbounded searchable collection or as a schema-constrained profile.
• Episodic-memory — records of specific past experiences. LangMem describes episodic memory as preserving "successful interactions as learning examples," capturing the full context of an episode: the situation, the reasoning that led to success, and why it worked. These become few-shot exemplars that guide future behavior.
• Procedural memory — knowledge of how to behave: the system prompt and evolving instructions the agent refines from feedback over time.

LangMem also distinguishes when memories are formed: in the hot path (extracted during the conversation, giving immediate updates at the cost of added latency) versus in the background ("subconscious" reflection after the conversation, finding patterns without slowing the live interaction).

Retrieval-augmented generation (RAG)

The dominant mechanism for connecting a model to long-term, factual knowledge is retrieval-augmented generation (RAG), introduced by Lewis et al. at NeurIPS 2020. The paper's framing is the cleanest way to think about agent memory in general: it combines parametric memory — knowledge baked into the model's weights — with non-parametric memory — an external index that can be inspected, updated, and grown without retraining. In the original work the parametric component is a pre-trained seq2seq generator (BART), and the non-parametric component is a dense vector index of Wikipedia accessed through a neural retriever (Dense Passage Retrieval, DPR).

The original paper introduced two formulations that differ in how often a fresh passage may be chosen: RAG-Sequence conditions on the same set of retrieved passages for the entire output sequence, while RAG-Token lets a different passage be used for each generated token. Lewis et al. reported state-of-the-art results on open-domain question answering and found RAG produced "more specific, diverse and factual" generations than a comparable parametric-only baseline.

Modern agentic RAG keeps the parametric/non-parametric split but typically implements retrieval at the prompt level rather than inside the decoder: a retriever finds relevant passages and the orchestrator prepends them to the context window before the model generates. The pipeline has two phases — an offline indexing phase that chunks source documents, embeds each chunk, and writes the vectors to a store; and an online query phase that embeds the user's question, finds the nearest chunks, and assembles the answer prompt.

flowchart TB
    subgraph Indexing["Indexing (offline)"]
        D[Source documents] --> C[Chunk into passages]
        C --> EM1[Embedding model]
        EM1 --> VS[(Vector store)]
    end
    subgraph Query["Query (online)"]
        Q[User question] --> EM2[Embedding model]
        EM2 --> SR[Similarity search]
        VS --> SR
        SR --> TOP[Top-k passages]
        TOP --> P[Assemble prompt:\nquestion + passages]
        H[Conversation history\nshort-term memory] --> P
        P --> LLM[LLM]
        LLM --> A[Grounded answer]
    end

Embeddings and vector stores

The component that makes semantic retrieval possible is the embedding: a vector (list) of floating-point numbers produced by a model such that the distance between two vectors reflects the relatedness of the two inputs. OpenAI states the principle directly — "small distances suggest high relatedness and large distances suggest low relatedness" — and recommends cosine similarity as the comparison metric. Because text that is semantically similar lands at nearby points in this vector space, retrieval can match a question to passages by meaning rather than by keyword overlap, which is why embedding search generalizes past exact wording.

A vector-store is the database that holds these embeddings and answers nearest-neighbor queries efficiently — returning the k passages whose vectors are closest to the query vector. The store is the non-parametric memory of a RAG system and the substrate of long-term semantic memory more generally; LangGraph, for instance, exposes semantic search over its memory store by configuring an embedding dimension and provider so that store.search ranks JSON memory documents by vector similarity.

Context engineering: managing a scarce window

Putting the pieces together, the umbrella discipline is context-engineering — what Anthropic defines as "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference." It is the successor to prompt engineering: prompt engineering optimizes the wording of a single instruction, while context engineering manages, on every turn of a long-running agent, the whole budget of tokens drawn from system instructions, tools, retrieved data, and message history. The goal is to keep the smallest set of high-signal tokens that lets the model take the right next action — directly counteracting context rot.

Anthropic describes several recurring techniques, each of which maps onto a piece of the memory picture above. The article groups the first three — compaction, structured note-taking, and sub-agent architectures — under context engineering for long-horizon tasks, while just-in-time retrieval is discussed separately under context retrieval and agentic search:

• Context-compaction — Anthropic defines this as "taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary." Compaction is a form of summarization here, and it is therefore lossy: Anthropic stresses that the art lies in choosing what to keep, since over-aggressive compaction can discard "subtle but critical context" whose importance only becomes apparent later.
• Structured note-taking (agentic memory) — the agent writes notes to a store outside the context window and retrieves them later, giving persistent memory with minimal token overhead. This is the write-path of long-term memory, driven by the agent itself.
• Sub-agent architectures — specialized sub-agents each work in their own clean context and return only a "condensed, distilled summary" to a coordinating agent, so the main window never accumulates the full detail of every subtask. See orchestration patterns.
• Just-in-time retrieval — instead of front-loading data, the agent holds lightweight identifiers (file paths, queries, links) and loads the underlying content into the window only when a step needs it, using tools. RAG is the canonical instance.

The throughline across all of these — short-term trimming, long-term stores, RAG, and compaction — is the same trade-off: the context window is the model's only sensory surface, it is finite, and good agent design is largely the art of deciding what gets to occupy it at each moment.