If you were to place an automobile from the early twentieth century next to a horse-drawn carriage, the comparison would make sense. The structure was similar, and the logic was too. The difference was simple: instead of the horse, there was an engine.
For a while, calling a car a “horseless carriage” was an honest description… But at some point, that stopped explaining what was in front of our eyes.
There was no single turning point. There was accumulation. Better brakes, suspension, new materials, electrical systems, ergonomics, urban infrastructure. Each improvement seemed small when viewed in isolation. Together, they changed the nature of the object.
The car ceased to be an improved carriage and became something else.
Something similar happened to us. The human being of today is not merely an improved primate. Between us and our ancestors, there was an accumulation of structure: writing, books, science, medicine, culture. We did not evolve only inwardly. We evolved because we learned to sustain knowledge over time.
It is this kind of silent, cumulative, structural change that is beginning to happen with language models.
For a long time, people tried to describe them as sophisticated autocomplete: a system that predicts the next token based on the available context. That metaphor was never entirely fair. Even the earliest transformers already captured complex relationships between words and context, far beyond simple autocomplete. Still, the image caught on. It was simple, and simplicity usually wins.
The problem is that it has aged badly.
Today, it is not only incomplete. It is beginning to hinder understanding of what is really changing. The joke outgrew its target… but the target changed position, and the joke simply no longer hits it.
Because the most important transformation is not only in the model.
It is in the infrastructure around it.
One of the most discreet pieces of this change goes by the name TurboQuant, which will probably be part of the structure of the next generation of language models.
At first glance, it looks like mere engineering: memory compression, bit reduction, efficiency. Something useful, but not very inspiring. As if we were discussing carburetors.
But that is not the point.
Language models operate on a resource that almost never appears in the conversation: active context. Everything that has been said must be kept, in some way, accessible while the model responds. This lives in a structure called the KV cache, the physical structure where the keys and values of tokens already processed during inference are stored, ultimately where the vector representations of the conversation reside.
This cache grows over time. And it grows fast.
The longer the interaction, the greater the memory and processing cost. At some point, the system has to choose between becoming too expensive or beginning to lose parts of what has already happened.
That is why these models seem to forget.
Not because they “do not know.” But because remembering, today, is too expensive.
TurboQuant enters precisely there.
It takes the vectors that make up this context and does something counterintuitive before compressing them. First, it applies a rotation in vector space. This rotation redistributes the information, preventing some dimensions from carrying more weight than others. In high dimensions, this makes the data more “uniform” and therefore easier to compress.
Then, each coordinate is quantized separately. Instead of using high-precision numbers, the system begins to represent each dimension with very few bits.
Up to this point, it would seem like merely aggressive compression.
What makes the method special is the next step. It corrects the distortion introduced by quantization so that the geometric relationships between the vectors — what actually carries meaning — are preserved.
In practical terms: the vector becomes much smaller, but it continues “pointing” in almost the same direction within semantic space.
And that is what matters.
Because, in language models, meaning does not lie in isolated words, but in the relationships between representations.
The result is simple to describe and difficult to underestimate. The system becomes able to store much more context using much less memory, with no perceptible loss of quality.
It is precisely at this point that the conversation ceases to be merely about compression and becomes about continuity. Saying that TurboQuant “saves memory” is correct, but insufficient. What it really does is make the persistence of context cheaper, and that has a far greater effect than it seems at first glance. Language models do not usually fail because they cannot produce plausible answers; they fail when they can no longer sustain, with the same sharpness, what has been built throughout the interaction. When context slips away, coherence slips away with it.
That is why longer conversations with LLMs often show that kind of wear any more attentive user has already noticed. The system starts well, seems to follow everything, but then loses references, contradicts earlier formulations, oversimplifies some important nuance, or stops perceiving the weight of what had been established before. This is often read as a cognitive limitation or hallucination, when it is often a technological contingency of support. The model remains capable, but it is no longer operating with the same density of past at its disposal.
When this bottleneck begins to be addressed, behavior changes in a way less spectacular than a viral demo, yet perhaps more important. The responses stop seeming so isolated from one another. The conversation gains more persistence, more internal connectedness, more ability to maintain a guiding thread without dissolving with each new block of text. This is not magic, nor a sudden metamorphosis of the model into another kind of entity. It is a very concrete consequence: the system becomes better able to carry what was already being built.
We humans do something analogous all the time, although in a much richer and more biologically rooted way. We do not remember everything with photographic precision, nor do we need to in order to maintain coherence. What we preserve are connections, relevancies, hierarchies, anchor points. That is what allows a conversation, an idea, or even a psychic life to have some recognizable continuity. In language systems, with all proportions duly observed and without hasty anthropomorphisms, improving the ability to sustain context produces a structurally similar effect: it does not create intelligence out of nothing, but it prevents it from dissipating so quickly.
That is why TurboQuant matters more than its name suggests. It is not a new model, not a charming interface, and not the kind of advance that usually circulates as public spectacle. Even so, it acts on one of the central bottlenecks of the contemporary generation of LLMs: the cost of maintaining context over time. And when that cost falls, the system’s limit shifts as well. What previously had to be discarded too early can remain; what was previously lost along the way can continue to have an effect.
For that reason, even though it is still possible to call LLMs autocomplete, the expression is becoming increasingly dated, and perhaps the most accurate thing to say is that it was already born dated. It captures a real aspect, but one that is far too small for the phenomenon. It is useful as caricature, not as description. In the same way that at some point it stopped making sense to look at a car and see it merely as a motorized carriage, it is beginning to make less sense to look at these systems and describe them merely as next-word predictors. As the infrastructure evolves, what is emerging is not only a system that responds better, but a system that is able to sustain more of what was at stake a few moments earlier.
That is where the change ceases to be merely quantitative. Because, when the capacity to continue improves enough, it changes not only the efficiency of the system, but the very way we begin to describe it.
Want to listen to the podcast?
Want to watch the video?
https://www.youtube.com/watch?v=SWCAx5xsA4I
Paper discussed…
TurboQuant:
Online Vector Quantization with
Near-optimal Distortion Rate
https://arxiv.org/abs/2504.19874
My project on…
Stochastic Consciousness:
Architectures for the Emergence of Meaning
in Context-Sensitive Language Systems
https://zenodo.org/records/19188165