Your LLM Has No Memory. Everything Else Is Engineering.

An LLM is stateless per call: each request is independent; no state survives inside the model. “Memory” is external engineering — what goes into the prompt, what is stored outside it, and when it is re-injected.

This post uses a voice assistant (speech in, transcript to model, text-to-speech out) as the running example.

1 Context window

Within one call, the model appears to remember because the full conversation is re-sent each turn.

The model reads one block of text per request.
The context window caps how much text fits; the transcript lives under that ceiling.
Turn n includes the system prompt and all prior turns in one request.

Code

SYSTEM = "You are a phone assistant. Two sentences max; replies are spoken aloud."

turns = [
    ("user", "Hi, call me Alex"),
    ("assistant", "Nice to meet you, Alex."),
    ("user", "How hard should tomorrow's ride be?"),
]
request = [{"role": "system", "content": SYSTEM}]
request += [{"role": r, "content": c} for r, c in turns]

print(f"{len(request)} messages, {sum(len(m['content']) for m in request)} characters sent")

4 messages, 145 characters sent

Every message is sent on each turn. Latency and cost grow with transcript length.

Compaction (context editing) replaces older turns with a summary when the window fills, rather than dropping the oldest turn outright (which often holds the most valuable facts).

Code

def compact(history, keep=4):
    """Replace all but the last `keep` turns with one summary message."""
    old, recent = history[:-keep], history[-keep:]
    gist = "; ".join(c for _, c in old)[:60]
    return [{"role": "system", "content": f"Earlier in this call: {gist}…"}] + [
        {"role": r, "content": c} for r, c in recent
    ]

long_call = turns * 8
print(f"{len(long_call)} messages -> {len(compact(long_call))} after compaction")

24 messages -> 5 after compaction

Compaction is lossy and dies when the call ends. Facts needed across sessions must be stored elsewhere.

2 Prompt slots

Each request has two injection points, not memory stores:

System prompt — fixed frame: persona, rules, constraints. In voice apps: short replies, no markdown, units spelled out.
User prompt — the current turn.

Cross-session facts must live outside the prompt and be injected back into a slot on the next call.

3 Long-term storage options

Four mechanisms; pick by size, shape, and lookup pattern:

3.1 RAG

For facts in a large, fuzzy, changing corpus (e.g., help-centre articles).

Embed document chunks; embed the query at call time; retrieve nearest neighbours by meaning.
Pattern: retrieval-augmented generation (RAG).
Cost: embedding call, extra tokens, occasional irrelevant chunks.

3.2 Key-value lookup

For small, discrete per-user facts (voice ID, name, language, timezone).

user_id → voice_id is faster and more reliable than vector search.
Belongs on the latency path (e.g., TTS voice selection).

3.3 Memory tool

For facts the schema did not anticipate.

Give the agent write/read tools pointed at durable storage.
The agent decides what to persist (e.g., tone preferences).
Requires curation: files grow, contradict, and go stale.

Code

import tempfile
from pathlib import Path

store = Path(tempfile.mkdtemp())  # a real deployment uses durable storage


def memory_write(user_id, text):
    """The agent's write tool: append a durable fact about this caller."""
    path = store / f"{user_id}.md"
    with path.open("a") as fh:
        fh.write(text + "\n")
    return f"wrote {len(text)} chars to {path.name}"


print(memory_write("u_8213", "- Prefers a calm, low-pitched voice; dislikes the default."))
print(memory_write("u_8213", "- Usually asks about cycling training."))

wrote 58 chars to u_8213.md
wrote 38 chars to u_8213.md

3.4 Fine-tuning

Rarely appropriate for personal facts.

Facts become part of model weights; not editable per user, not inspectable, not deletable without retraining.
Use for behavior and style, not per-user state.

4 Compaction and memory tool ordering

Compaction rewrites older turns into a summary. If compaction runs before the memory tool writes durable facts, information is lost.

Write durable facts before compaction lands.
RAG needs a vector database for embedded corpus search.
Prompt caching is a latency/billing optimization over a stable prefix, not memory.

5 Session assembly

Session two starts with an empty window. The assistant “knows” the caller because code assembles context before the first token:

Code

voices = {"u_8213": "bm_george"}  # key-value store: user_id -> Kokoro voice pack

remembered = (store / "u_8213.md").read_text()
system_prompt = f"{SYSTEM}\n\nWhat you know about this caller:\n{remembered}"

print("TTS voice:", voices["u_8213"])
print("---")
print(system_prompt)

TTS voice: bm_george
---
You are a phone assistant. Two sentences max; replies are spoken aloud.

What you know about this caller:
- Prefers a calm, low-pitched voice; dislikes the default.
- Usually asks about cycling training.

Key-value lookup for voice
Memory file read into system prompt
Empty window filled turn by turn; RAG may run mid-call for corpus questions

6 Kokoro TTS

bm_george names a voice pack in Kokoro-82M: 82M parameters, Apache-2.0, StyleTTS 2 architecture, ISTFTNet vocoder, 54 voices, ~327MB.

Personalizing voice costs one string in a key-value row; the TTS model runs locally.

Audio clips were generated ahead of render by src/make_audio.py against Kokoro weights; only greeting text and voice ID differ:

Session 2 uses name from the memory file and voice from the key-value store; neither came from the LLM.

7 Mechanism comparison

Concept	What it stores	Where it lives	Persists across sessions?	Best for
Context window / conversation history	Transcript of one session	The request, every turn	No	Coherence within one call
System prompt	Persona, rules, injected memory	Top of every request	No — you re-inject it	Session-wide behaviour: “keep replies speakable”
User prompt	The current turn	The request	No	Whatever the user just said
Compaction / context editing	A summary replacing older turns	The request, rewritten	No	Sessions that outgrow the window
RAG	Documents, as embeddings	Vector database over your corpus	Yes	Large fuzzy corpora searched at call time
Key-value store lookup	Small discrete facts, by key	Any ordinary datastore	Yes	Per-user settings: voice ID, name, language
Memory tool (agent-managed files)	Whatever the model judged durable	Files it reads and writes via tools	Yes	Preferences no schema anticipated
Fine-tuning	Patterns baked into weights	The model itself	Yes, but not editable	Behaviour and style, not personal facts

8 Debugging

The model never remembers; failures are assembly or storage issues:

Forgetting within a call → compaction dropped information
Forgetting between calls → nothing was stored
False statements → retrieval miss or stale stored fact

9 References

How Does RAG Work?
Kokoro-82M — text-to-speech model used for demo audio