Several people have written to me recently to point out that my theory of intelligence is very similar to that of Jeff Hawkins and the folks at Numenta, a company Hawkins co-founded in 2005 with Dileep George (1) and Donna Dubinsky. The amazing thing is that Hawkins and I have arrived at a similar understanding of the brain via very dissimilar routes. Hawkins draws his inspiration from his knowledge of neuroscience and I get mine mostly from my interpretation of ancient Biblical metaphorical texts. I just finished reading a portion of Numenta's HTM Cortical Learning Algorithms (pdf). I think I now understand enough about the theory underlying HTM to form an educated opinion. Let me come right out and say that I think that Numenta's overall philosophy with regard to the function and operation of the cortex is basically sound but their current design is flawed. Furthermore, there is no way I can cooperate with Hawkins, Numenta or anybody associated with their HTM technology. I explain below.
Temporality, Hierarchy and Prediction
In 2005, Hawkins published his book, On Intelligence, in which he revealed his theory of intelligence. Based on the parts of the book that I've read, I think that Hawkins makes a convincing case for his ideas. He argues that the ability to anticipate the future and to learn sequences of patterns is the basis of intelligence. He further argues that cortical learning is universal in the sense that the neocortex uses the same learning and prediction algorithms to process visual, tactile or auditory information. What is important, he maintains, is the temporal relationships that can be inferred from parallel streams of discrete sensory data. Those relationships can be learned and stored in a hierarchical memory structure. I think others have made similar arguments before but Hawkins is the first to pin it down in an easy to read book. I essentially agree with Hawkins' position on intelligence. As some of my readers already know, I have been saying pretty much the same thing for many years.
After reading Dileep George's interesting PhD thesis, How the Brain Might Work: A Hierarchical and Temporal Model for Learning and Recognition (pdf), I can't help but conclude that academics, in general, love to complicate things just for the sake of complexity. It might also be a way to impress one's peers. I mean, unless one can show one's mathematical prowess, regardless of its relevance to one's thesis, one can forget about becoming a doctor in philosophy. Dr. George apparently believes that mathematics, in the form of Bayesian belief propagation equations, is essential to perceptual learning and recognition. I think he is mistaken. I believe that a fully functional artificial brain can be implemented with nothing fancier than basic arithmetic operations. What Dr. George calls belief propagation can be done simply by having every node at the input level in a hierarchy trigger a recognition signal whenever its input signals add up to a majority. This works for both sequential and concurrent pattern recognition. Repeat the same method at every level in the hierarchy until an entire branch is activated, indicating that a certain object has been recognized. By the way, this is what Zechariah was alluding to in his little occult book when he wrote:
[...] Behold, I will bring forth my servant the branch.The books of Zechariah and Revelation use terms like filthy garments or iniquity to refer to noisy and incomplete data. The ability to work with corrupted (filthy) data is what pattern recognition experts call pattern completion. It is a natural consequence of an intelligent system's predictive capability. It is a very powerful and effective mechanism, yet extremely simple. I know this because I use it in Animal's tree of knowledge. It is powerful because it allows an intelligent system to recognize sensory patterns even in situations where only partial or noisy information is available. Again, it can all be done with simple arithmetic operations. No need for any fancy math. In my opinion, mathematics is very much overrated. I have found that it is not needed in almost all cases.
[...] And I will remove the iniquity of that land in one day.
[Why do I bring up my work on Biblical symbolism in this critique? Because, as seen in the last section below, it is a crucial part of the point that I want to make.]
In spite of Hawkins' claim that Numenta's HTM technology is universal in that it can handle any kind of sensory learning, Numenta's current focus is restricted to visual learning and recognition. So it is no surprise that their literature frequently mentions spatial patterns as seen below:
An HTM region learns about its world by finding patterns and then sequences of patterns in sensory data. The region does not “know” what its inputs represent; it works in a purely statistical realm. It looks for combinations of input bits that occur together often, which we call spatial patterns. It then looks for how these spatial patterns appear in sequence over time, which we call temporal patterns or sequences.I think that using the term "spatial patterns" to refer to concurrent inputs is distractive and misleading because, from the point of view of the learning algorithm, there is no such thing as a spatial signal. It is all temporal, in the sense that the only relationships that can exist between discrete signals are simultaneity and sequentiality. The problem is that Numenta's approach forces them to determine in advance the boundaries of those spatial patterns. For example, in the case of vision, the designer is forced to select a small square area of pixels to act as inputs for the low-level concurrent pattern learner. This is a mistake, in my opinion, primarily because important information can be overlooked as a result of using arbitrarily restrictive boundaries. One wonders how Numenta would set boundaries for concurrent audio patterns.
A five-node sequenceFurthermore, I disagree with Numenta's idea that concurrent patterns (depicted as multi-input nodes in the above diagram) must be learned before sequential patterns. In my opinion, the suitability of input signals to a given concurrent pattern (a node in a particular sequence) is not determined solely by their simultaneity but also by whether or not the node belongs to the sequence. In other words, frequency is the main fitness criterion for temporal learning. I have written about this before.
Note: I have since changed my mind on this. I now believe that patterns must be learned independently of sequences. (10/23/13)
Christian AI Versus Atheist AI
Hawkins is very dismissive of those he disagrees with although I don't fault him for that. I think that, right or wrong, we should all have the courage to stand for what we believe in. Hawkins essentially dismisses the entire symbolic approach to AI taken during the latter part of the twentieth century by early AI pioneers like Herbert Simon, Marvin Minsky, John McCarthy and many others. He does not come right out and say that the symbolic AI gang is out to lunch (which they are, in my view) but it is obvious that his ideas on intelligence leave no room for all that symbolic nonsense. However, even though Hawkins preaches that atheists should not go around proclaiming their atheism for fear of antagonizing religious folks (the majority), he himself makes no bones of the fact that he is an atheist and an evolutionist. He wears the label proudly. Those of us who believe that the universe was intelligently designed and created are all a bunch of idiots in his view.
One of the reasons that I am bringing this up is that someone suggested in a recent comment that I should embrace Hawkins' technology and consider partnering with others on AI projects based on Numenta's HTM. The truth is that I have indeed thought of doing so in the past but I have since decided that Hawkins' virulent atheism turns me off. Not just because I am a Christian, but because Hawkins preaches that the best way to promote atheism is for atheists to accomplish great technological and scientific feats in the name of atheism. I, by contrast, I am of the opinion that the best way to advance Christianity is for Christians to accomplish great technological and scientific feats in the name of Christianity. And, like Hawkins, I put my money where my mouth is. I am already claiming that I obtained almost all of my understanding of the brain and intelligence from deciphering a few ancient Biblical metaphorical texts. I am even willing to go out on a limb and claim that, based on my interpretation of the ancient texts, I understand enough about the brain to write a computer program that, given adequate computing resources, will learn to behave intelligently in a manner similar to human beings.
It may indeed be possible to study human behavior and the brain's biology and use one's findings to eventually figure out how human intelligence works but, in my opinion, this approach will take a very long time. It is the approach that I originally took when I first became interested in AI. It did not get me very far because neuroscience is too chaotic. Finding relevant information by browsing the literature is like searching for the proverbial needle in the haystack. I have since found what I believe to be a much better and faster way to solve the AI problem. I realize that my approach is rather unconventional but I have researched it on and off for more than eight years and I am convinced now more than ever that I am on the right track. After all, how is it possible that my findings are in basic agreement with neuroscience and Hawkins's own ideas?
There can be no doubt that my world view and my approach to AI is anathema to atheists like Hawkins. Paul Z. Myers, an atheist biologist who seems to have a major bone to pick with Christians, once wrote an entire blog article to ridicule me. But there is a flip side to this coin. I am just as dismissive of Hawkins and his atheist colleagues in the scientific community as they are of Christians like me. Hawkins is one of those deeply religious people (we are all religious, especially if we are convinced that we are not) that I have taken to calling dirt believers. Essentially, a dirt believer is a person who is convinced that matter (dirt) sprang out of nothing all by itself and that life sprang out of dirt all by itself. I think this is an excruciatingly idiotic view (for reasons that I will not go into because they are beyond the scope of this post). Consequently, there is no way I will cooperate or partner with either Jeff Hawkins, Dileep George or any other atheist in intelligence research or pretty much anything else. The way I see it, what we have here is a battle between the atheist AI and the Christian AI. May the best religion win.
Missing Pieces in Numenta's Memory Model
1. Dr. Dileep George has left Numenta to form his own AI company, Vicarious Systems. Like Hawkins, Dr. George is also an atheist.