On Intelligence

On Intelligence Plot Summary

Introduction

How does the human brain truly work? Despite centuries of scientific inquiry, the fundamental principles of intelligence have remained elusive. Traditional approaches to artificial intelligence and neural networks have largely failed because they've focused on replicating behaviors rather than understanding the brain's core operational framework. This disconnect has led to a peculiar situation where we've accumulated mountains of neuroscientific data without a coherent theory to make sense of it all. The memory-prediction framework offers a revolutionary perspective on intelligence. This theory proposes that intelligence isn't about processing inputs to generate outputs, but rather about memory systems that continuously make predictions based on stored patterns. The neocortex, with its uniform structure and hierarchical organization, implements a common algorithm across all sensory and cognitive functions. By understanding how the brain builds models of the world, stores sequences of patterns, and makes predictions through invariant representations, we can finally grasp the nature of intelligence itself—and potentially build machines that think in fundamentally the same way humans do.

Chapter 1: The Limitations of AI and Neural Networks

Artificial Intelligence has historically been defined by its focus on behavior rather than understanding. Since the inception of AI in the 1950s, researchers have attempted to program computers to produce intelligent behavior without first addressing what intelligence actually is. This behavioral focus stems from Alan Turing's influential idea that a machine could be considered intelligent if it produced responses indistinguishable from human answers. The assumption was that with sufficient processing power and programming sophistication, computers would eventually become intelligent. This approach fundamentally misunderstands the nature of intelligence. The brain doesn't compute answers to problems; it retrieves them from memory. While computers can perform billions of operations per second compared to neurons' modest few hundred, the human brain can solve complex problems like recognizing a cat in an image in less than half a second. This is possible because our brains operate on a different principle—what can be called the "one hundred step rule." Given the slow speed of neurons, complex perceptual tasks must be accomplished within approximately one hundred neural steps, not the billions of computational steps required by conventional computing approaches. Neural networks attempted to address some of AI's shortcomings by mimicking how neurons connect and learn. However, most neural networks evolved into simplified systems with three layers of artificial neurons—input, hidden, and output—that still focused on mapping inputs to outputs. These networks lacked crucial features of real brain function: they didn't incorporate time as a fundamental dimension, they didn't utilize feedback connections (which in real brains outnumber feedforward connections by a factor of ten), and they didn't replicate the hierarchical architecture of the neocortex. Perhaps the most damning critique of traditional AI came from philosopher John Searle's "Chinese Room" thought experiment. Searle imagined a person following a complex set of rules to respond to Chinese messages without understanding Chinese. Similarly, computers following AI programs can produce intelligent-seeming behaviors without actually understanding anything. Understanding isn't about behavior—it's about predicting what will happen next based on a model of the world. Real intelligence requires memory and prediction working together in a way fundamentally different from conventional computing architectures. The failure of traditional AI approaches stems from their reliance on an intuitive but incorrect assumption: that intelligence is defined by observable behavior. This intuition has proven to be a major obstacle to discovering the true nature of intelligence, just as intuition once incorrectly suggested that the Earth was stationary at the center of the universe. To build truly intelligent machines, we need to start with a correct understanding of what intelligence actually is—a memory system that makes predictions.

Chapter 2: Understanding the Hierarchical Structure of the Neocortex

The neocortex is a remarkably uniform sheet of neural tissue about 2 millimeters thick, organized into six distinct layers. Despite appearing similar throughout, this thin sheet is responsible for nearly all aspects of human intelligence—perception, language, mathematics, art, music, and planning. What makes this structure so powerful isn't its complexity but rather its organizational principles, particularly its hierarchical arrangement of regions that mirror the hierarchical structure of the real world. This hierarchical organization was first comprehensively described by Vernon Mountcastle in his revolutionary 1978 paper. Mountcastle made the startling observation that despite handling different types of information (vision, hearing, touch, etc.), all regions of the neocortex appear to be performing the same basic operation. The visual, auditory, and motor areas of the cortex all have the same layered structure, cell types, and connectivity patterns. What makes them specialized isn't their internal functioning but their external connections to different sensory organs and other brain regions. The cortical hierarchy is arranged so that information flows both up and down through interconnected regions. Sensory information enters through primary sensory areas like V1 (primary visual cortex) at the bottom of the hierarchy. As information moves upward, it undergoes transformations that capture increasingly abstract relationships. For example, cells in V1 respond to simple features like oriented lines in specific locations, while cells higher up in IT (inferotemporal cortex) respond to complex objects like faces, regardless of their position, size, or orientation. This progression from specific to invariant representations occurs in all sensory systems. Crucially, information doesn't just flow upward in this hierarchy—there are massive feedback connections flowing downward as well. In fact, there are approximately ten times more connections going down the hierarchy than going up. These feedback connections carry predictions from higher levels to lower levels, allowing the brain to constantly check its model of the world against incoming sensory data. When predictions match sensory input, we understand what we're perceiving. When they don't match, the mismatch travels up the hierarchy until some higher region can make sense of the unexpected input. The power of this hierarchical structure becomes evident in how we interact with the world. When you recognize a familiar melody played in a new key, or identify a friend's face from a new angle, your cortex is using invariant representations stored in its hierarchical memory to make predictions about what you're experiencing. The combination of feedforward sensory information and feedback predictions creates our rich experience of a stable, comprehensible world. This same process allows us to recognize patterns in increasingly abstract domains, from physical objects to language to mathematics to social relationships.

Chapter 3: Memory and Invariant Representations

Memory in the neocortex functions fundamentally differently from computer memory. While computers store information exactly as it's presented, retrieving it perfectly upon command, the neocortex stores patterns in ways that capture the essential relationships in the world, independent of specific details. This capacity to form what are called "invariant representations" is central to how our brains understand the world and make predictions about it. The cortex stores sequences of patterns that unfold over time. Think about how you recall your home—you can't think of every room simultaneously. Instead, you mentally walk through it room by room. Similarly, you can't recall a song all at once—you experience it as a sequence of notes unfolding over time. Even seemingly static memories like recognizing a friend's face involve sequences of eye movements and patterns that activate sequentially in your brain. This sequential nature of memory reflects how we experience the world through time. Another crucial property of cortical memory is its auto-associative nature. Auto-associative memory allows you to recall complete patterns when given only partial or distorted inputs. If you see just your friend's eyes peeking over a fence, you can recognize their entire face. If you hear a familiar melody with some notes missing, your brain automatically fills in the gaps. This completion happens automatically and unconsciously as part of normal perception. The brain is constantly filling in missing information based on learned patterns. The most remarkable feature of cortical memory is how it creates invariant representations. When you recognize a melody played in a different key, or identify a friend's face from a different angle or in different lighting, your brain is using invariant representations. For example, when you look at your friend from different distances, the image on your retina changes completely in size, yet you perceive the same person. Your brain has stored the invariant properties of your friend's face—the relationships between features, regardless of size, angle, or lighting conditions. To make specific predictions from these invariant memories, the brain combines its stored knowledge with current context. When listening to a familiar melody in a new key, your brain predicts the next note by combining the invariant representation of the melody (the pattern of intervals) with the specific context (the last note you heard). Similarly, to predict exactly what you'll see when your friend turns around, your brain combines its invariant memory of their face with the specific current view. This ability to merge general knowledge with specific context allows the brain to make precise predictions about constantly changing sensory inputs. These properties of cortical memory—storing sequences, auto-associative recall, and invariant representations—create a memory system perfectly designed for prediction. Rather than computing answers to problems, our brains constantly use memories of past experiences to predict what will happen next, forming the foundation of intelligence itself.

Chapter 4: Prediction as the Core of Intelligence

Intelligence is fundamentally about prediction—the ability to anticipate what will happen next based on stored memories. This insight transforms our understanding of what intelligence actually is. Rather than being defined by behaviors or outputs, intelligence emerges from the brain's continuous efforts to predict its sensory inputs. Every moment of your waking life, your neocortex is making predictions about what you will see, hear, and feel next. Consider a simple experiment: imagine someone secretly alters your front door while you're away—perhaps moving the doorknob an inch to the left or changing its weight. When you return home and interact with the door, you'll immediately notice something is wrong. This instant recognition happens because your brain makes precise predictions about what you expect to feel, see, and hear as you approach and open the door. When those predictions aren't met, the mismatch immediately captures your attention. Prediction is so pervasive in our cognitive processes that we rarely notice it until our expectations are violated. Predictions operate across multiple levels of the cortical hierarchy simultaneously. Lower levels predict specific sensory details, while higher levels predict more abstract patterns and relationships. When you're reading a book, your predictions range from the shapes of upcoming letters to the meanings of words to the narrative arc of the story. These multi-level predictions happen in parallel across all your senses. When you ride a bicycle, your brain predicts the visual scene ahead, the feel of the handlebars, the pressure on your feet, the sounds of the chain—all coordinated into a unified sensory experience. This predictive capability allows us to understand the world at a fundamental level. To know something means to be able to make predictions about it. When you understand physics, you can predict how objects will fall. When you understand another person, you can predict how they'll behave in different situations. Even abstract knowledge involves prediction—mathematical understanding means being able to predict the result of applying mathematical operations. This is why intelligence tests often ask you to predict the next number in a sequence or the missing element in a pattern. The relationship between prediction and behavior is subtle but profound. While behavior is ultimately important for survival, it's prediction that guides intelligent behavior. In fact, behavior and prediction are two sides of the same process. When you predict seeing your arm move, this prediction itself causes the motor commands that make the prediction come true. You think first, which causes you to act to make your thoughts become reality. This integrated system of prediction and action, implemented throughout the cortical hierarchy, creates the remarkable flexibility and effectiveness of human intelligence.

Chapter 5: How the Cortex Creates and Uses Mental Models

The neocortex builds and maintains a model of the world through its hierarchical memory structure. This model reflects the nested, hierarchical organization of reality itself—objects are composed of subobjects, which themselves contain smaller components, forming a natural hierarchy. Your cortex discovers and captures this structure automatically through experience, storing patterns and their relationships in its hierarchical memory system. Each region of the cortex learns and stores sequences of patterns, developing what can be thought of as "names" for sequences it recognizes. These names are passed to higher regions in the cortical hierarchy. As information moves up the hierarchy, stable patterns emerge that represent invariant aspects of the world. Higher regions maintain representations of larger-scale structures and relationships, while lower regions deal with rapidly changing details. When you walk into your living room, higher regions maintain the representation "I'm in my home" while lower regions process specific sensory details of the moment. The cortex implements this hierarchical memory through its unique six-layered structure and columnar organization. Cortical columns are small units of cells that extend vertically through all six layers. Each column can become active when it recognizes specific input patterns, and columns learn to predict when they should become active based on context. When a prediction is correct, the column sends a stable pattern (a "name") upward to higher regions. When a prediction is violated, it sends information about the unexpected pattern upward. This architecture allows predictions to flow down the hierarchy and unexpected patterns to flow upward until they can be understood. The biological implementation involves specific arrangements of cells in different cortical layers. Layer 4 receives input from lower regions. Layers 2 and 3 project to higher regions, while layer 6 sends feedback to lower regions. This layered structure provides the physical framework for the bidirectional flow of information essential to prediction. When patterns are unexpected, information propagates upward until some region can interpret it. When patterns are recognized, predictions flow downward, preparing lower regions for what they should expect next. This system becomes more powerful through learning. Initially, a child's brain forms memories of simple patterns in higher cortical regions. With repeated exposure, these memories move to progressively lower regions in the hierarchy. For example, when first learning to read, recognizing individual letters requires conscious effort. Eventually, letter recognition moves to lower cortical regions, freeing higher regions to recognize whole words and phrases. This process allows experts to perceive patterns invisible to novices—the chess master sees strategic positions while the novice sees individual pieces. The memory-prediction framework explains how we build mental models of the world through experience. These models allow us to recognize patterns, fill in missing information, resolve ambiguities, and predict what will happen next. By capturing the hierarchical structure of reality in its own hierarchical organization, the cortex creates a system perfectly designed for understanding and navigating our complex world.

Chapter 6: Building Truly Intelligent Machines

Creating machines with genuine intelligence requires a fundamentally different approach than traditional artificial intelligence or neural networks. Rather than programming behaviors or training simplified neural networks, we must build systems that implement the cortical algorithm of memory and prediction. This means designing hierarchical memory systems that learn from experience, form invariant representations, and make predictions about their world. The challenges in building such machines are substantial but solvable. We need memory systems with sufficient capacity to store complex world models. The human neocortex has approximately 30 trillion synapses; replicating this capacity in silicon would require significant but achievable memory resources. We also need to solve the connectivity problem—how to enable the massive number of connections between artificial neurons that real cortical systems possess. While these challenges are real, there are no fundamental obstacles to building brain-inspired intelligent machines. Importantly, these intelligent machines won't resemble the humanoid robots of science fiction. There's no reason for intelligent machines to look, act, or sense like humans. What makes a system intelligent is its ability to form a model of its world and make predictions, not its physical embodiment. These systems could have entirely different sensory capabilities than humans—perhaps weather sensors distributed across a continent, or molecular sensors that perceive protein folding. They could think a million times faster than humans and possess vastly larger memories. These capabilities would allow them to solve problems beyond human capacity. The potential applications are profound. In the near term, intelligent machines could transform domains where traditional AI has struggled, such as understanding natural language, processing visual scenes, or navigating complex environments. More revolutionary applications will emerge as these systems develop. Weather-sensing intelligent systems might discover new climate patterns. Medical systems might find relationships in disease data invisible to human researchers. Systems with exotic senses might solve problems in physics, chemistry, or mathematics that have stumped human scientists. Unlike some technological advances that pose existential risks, intelligent machines based on the memory-prediction framework would be relatively safe. They wouldn't have human emotions like fear, greed, or ambition unless deliberately programmed with them. They would be powerful tools for expanding human knowledge and capabilities, not competitors or threats. Building such machines doesn't require replicating human consciousness or copying human brains; it requires implementing the cortical algorithm in silicon. The timeline for this technology remains uncertain. Progress may seem slow initially but could accelerate rapidly once the correct approach takes hold. With the memory-prediction framework as a guide, researchers and engineers can focus their efforts on implementing the principles that actually create intelligence rather than attempting to simulate behaviors. By understanding intelligence correctly—as a memory system that makes predictions—we can finally build machines that truly think.

Summary

The memory-prediction framework represents a profound shift in our understanding of intelligence. At its core, intelligence is not defined by behavior but by the ability to make predictions based on stored memories. The neocortex, with its uniform structure and hierarchical organization, implements a common algorithm across all its regions: it remembers sequences of patterns, creates invariant representations, and constantly predicts what will happen next. This framework explains how we perceive a stable world despite constantly changing sensory inputs, how we understand abstract concepts, and how we can imagine future scenarios. This new understanding of intelligence opens remarkable possibilities. By implementing the cortical algorithm in silicon, we can create intelligent machines with capabilities complementary to human intelligence—machines that think faster, possess larger memories, and perceive through novel senses. These systems won't be the humanoid robots of science fiction, but they will transform how we explore and understand our world. The memory-prediction framework doesn't diminish the wonder of human intelligence; rather, it deepens our appreciation for the elegant solution evolution discovered and points the way toward extending intelligence beyond biological constraints.

About Author

Loading...

Jeff Hawkins

Read more

Related Books by Jeff Hawkins

Loading...

Phantoms in the Brain

V.S. Ramachandran

Probing the Mysteries of the Human Mind

★★★★☆4.3

Loading...

A Thousand Brains

Richard Dawkins

A New Theory of Intelligence

★★★★☆4.1