Intelligence explosion is real

“Intelligence explosion” is the idea that once we create some form of artificial intelligence that is instrumental to creating an even smarter artificial intelligence, it will lead to a rapid and exponential increase in intelligence. The concept was first proposed by mathematician I.J. Good in 1965, who suggested that an ultraintelligent machine could design ever better machines, leading to a runaway growth in capability that would leave human intelligence far behind.¹

This concept is often associated with the idea of a technological singularity: the point at which the growth of intelligence becomes uncontrollable and irreversible. I want to stress that an intelligence explosion is not automatically equivalent to a singularity: the latter is a possible consequence of the former, but not necessarily the only one.

The debate online is, as is often the case, very polarized. Some claim singularity is inevitable and will happen in the next few years; others dismiss it as science fiction. I’m going to lay out my definitions of intelligence, consciousness, and singularity, and then explain why I believe intelligence explosion is already underway, just not in the form most people imagine.

What is intelligence then?

A few common definitions:

• “The ability to acquire and apply knowledge and skills” - Oxford Dictionary
• “The ability to learn, understand, and make judgments or have opinions that are based on reason” - Merriam-Webster Dictionary
• “The aggregate or global capacity of the individual to act purposefully, to think rationally, and to deal effectively with his environment” - Wechsler²
• “Goal-directed adaptive behavior” - Sternberg & Detterman³

Notice a few things:

1. All definitions imply at least interaction of a system with its environment.
2. These are not definitions of “human intelligence” but of intelligence in general, applicable to any system.
3. Intelligence is not a binary property but a spectrum.

Is consciousness the same as intelligence?

No. Consciousness requires at least temporal continuity of experience and some form of self-model, an internal representation of oneself as a distinct entity persisting through time. Whether LLMs possess the latter is debatable, but they clearly lack the former: each inference call is stateless, with no persistent thread of experience connecting one interaction to the next.

It’s real, and it’s already happening

The intelligence explosion is not an “autonomous AGI that recursively self-improves without human intervention.” It’s something more mundane and, precisely because of that, more real.

Humans are filling in the gaps. We build systems that iterate on new versions of themselves. They inevitably make mistakes and fail, and we’re there to fix them, to patch them, to teach them, and to make them better. The next generation of models is trained partly on data generated or curated by the previous generation; humans close the loop where automation breaks down.

The fact that the previous paragraph doesn’t sound like a big deal should make the reality of intelligence explosion self-evident.

Intelligence is a property of a system, and in this case the system is partially made of humans and partially made of machines. This is a societal intelligence explosion, not a purely artificial one.

The obvious counterargument

If humans are in the loop, then the rate of improvement is bounded by human cognitive and work-time bandwidth. This is true… for now. The bottleneck is real and I’ll discuss its implications in a future post. But even a human-bounded explosion is still an explosion in historical terms: the feedback cycle between human researchers and AI systems is compressing what would have been decades of progress into years. The growth rate doesn’t need to be infinite to be profoundly disruptive; it only needs to outpace society’s ability to adapt.

Anthropic says not even their latest huge Mythos model can cut the new models research time in half⁴. But it’s speeding up for sure.

One more interesting point is how profound is the perceived performance difference between benchmarks and real-world usage. Benchmarks scores improved at an almost linear pace in the last few years, percived usefulness obviously does not. As many keep saying online, a small portion of a noisy exponential curve may perfectly look like a linear trend. I’m quite sure this is the case here, we’ll see if time proves me wrong.

LLM Development Is a Form of Self-Supervised Learning

I couldn’t understand why people keep denying that LLMs exhibit a form of intelligence, and then I realized that these people are building reasoning on an oversimplified picture of the training process.

I’d like these people to understand this:

LLMs are NOT trained to predict the next token. That’s just a pretext task.

Next-token prediction is a form of self-supervised learning. You cannot teach a model to “reason” directly, because you don’t have a formal specification of what reasoning is. So instead you train it to predict text. If the model becomes truly excellent at this task, it must learn to capture logical structure, causal relationships, and inferential patterns — otherwise it cannot correctly predict the next token across the full distribution of human-generated text, no matter how much data you provide.⁵

But next-token prediction is not the final product. On some tasks, text-only models are naive or outright confabulate. So you incorporate additional modalities, run reinforcement learning passes (RLHF, RLAIF, RLVR, and many variants you may look up) to improve task-solving ability, perform adversarial training, and iterate. The result is a model with more coherent output and stronger reasoning capabilities. You use it for data augmentation and filtering, and the cycle begins again.

The whole process is not training for text generation. Text generation is the substrate; the emergent capability, imperfect, partial, but undeniably present is a form of general reasoning.

Note that self-reinforcement learning works great in many cases and we know it. That’s basically how Meta, Google and others, for almost all practical applications, solved computer vision. It’s not magic, it’s math. If a training target requires an authentic understainding or reasoning of some sort, deep neural networks will learn to perform it.

On the other hand, it astonishes me that LLMs are capable of any kind of reasoning at all. Intelligence is an emergent property that can arise in unexpected ways from diverse physical substrates, this should be a source of profound philosophical debate. For sure the usefulness of these systems far exceeds what one could predict from a bare description of their architecture.