AI Vocabulary

Memory: why is a long context window not enough?

The context window is what a model can read at once. Memory is what a system keeps from one time to the next. They are distinct: an enormous window allows a very long document to be processed, it does not allow anything to be remembered from last week.

What a context window is

It is the amount of text a model can take into account to produce an answer: your request, the documents you attach, and the exchange in progress. Recent models accept the equivalent of several hundred pages, which was unthinkable not long ago.

That capacity is advancing fast, it is spectacular, and it features in every announcement. Rightly so: it allows a whole contract to be submitted instead of being cut up.

Why that is not memory

Because the window empties. When the exchange ends, everything it held disappears. Next time, the system starts from nothing: it knows neither that you have worked on this matter before, nor what you concluded.

The consequence is one every practitioner has noticed without naming it: the cost of establishing context is paid in full every time. Gathering the documents, explaining the situation, restating the positions — three hours the first time, three hours the second.

And that cost does not spread across later uses, contrary to what one would expect of an investment. It repeats identically, which explains why most users return to short tasks after two or three attempts.

A useful distinction follows: the window is a reading capacity, memory is a retention capacity. Increasing the first has never produced the second, and no future progress in models will, because they are not the same problem.

What real memory presupposes

Something that exists outside the model and outlives it: a place holding the matter, the decisions taken, their reasons, the established terminology. The model draws on it when interrogated, and deposits there what is worth keeping.

That is a property not of the model but of what surrounds it. A model with a gigantic window backed by nothing has no memory; a more modest model backed by a layer that retains does.

What it does not solve

Memory holds only what has been entrusted to it. A trade-off taken in a meeting and never recorded will not appear there, whatever the quality of the architecture.

It also raises a question the window did not: what should be kept, and for how long? Keeping everything produces an unusable volume; keeping only conclusions loses the reasons, which is precisely what one meant to preserve.

And it makes nothing more reliable in itself. A mistaken decision faithfully preserved will be served up unchanged for three years.

Why it matters to a lawyer

Because this is the point on which product announcements are most misleading without lying. An extended context window is a real achievement, and it is presented in a way that suggests the system remembers.

The test fits in one question: if I come back to this matter in three weeks, what does it still know? If the answer is nothing, there is no memory, whatever the size of the window.