A man and a dog named Sidney walk into a bar. The bartender tells the man that he cannot bring a dog into the bar. “But,” says the man, “this is a talking dog. Not only that, he thinks, reasons, and evaluates.” The bartender says “if any of that is true, then I will let him stay.” The man turns to the dog and asks, “If I don’t shave for three days, how does my face feel?” The dog says “rough.” The man asks, “What is the top of a house?” The dog says “roof.” The bartender still does not believe them, so the man asks the dog, “Who is the best baseball player of all time?” The dog answers “Ruth.” With that last answer, the bartender throws the man and dog out of the bar. As they are leaving, the dog turns to the man and says, “Should I have said Ty Cobb?”
This joke is a metaphor for the current situation with large language models, also known as GPT models, ChatGPT, and GenAI. Let’s put aside the punchline for a minute and look at the bartender’s reasoning. The dog produced the correct answers to questions about grooming, construction, and baseball. But, the bartender wondered, did those answers indicate that the dog knew things about grooming, construction, and baseball? If they did, then this dog would be a super-canine with capabilities far beyond the ordinary dog. On the other hand, dog barks often sound like “rough,” “roof,” and “Ruth.” How can I tell whether the dog knows how to answer these questions versus just knows to produce these sounds in response to questions?
There was a real-life version of this dog joke, but it involved a horse, not a dog. Clever Hans was a horse who lived around the start of the 20th Century, and who appeared to be able to perform mathematical and other cognitive tasks.
People would ask the horse questions and the horse would tap its hoof to provide the answer. Hans and his owner, Wilhelm von Osten, toured Germany exhibiting the horse’s skills. In 1907, the psychologist, Oskar Pfungst discovered that Hans did not know the answers to the questions he was being asked. Rather, the horse had learned to tap his hoof until he detected involuntary cues from von Osten or others that he should stop—meaning that he had produced the correct answer. If none of the people around the horse knew the answer to a question, then, Pfungst discovered, neither did the horse. Three points to bring up.
(1) The public wanted to believe that Hans was cognitively sophisticated. Interest in and acceptance of the cognitive capacities of the horse were widespread.
(2) Hans was clever, but not in the way that he was credited. The problem that Hans was solving was different from the nominal problem that his audience thought he was solving.
(3) It took careful analysis and experimentation to find out what the horse was doing and not doing.
That is where we are today with large language models. They appear to show evidence of powerful cognitive processes, such as reasoning, creativity, narrative understanding and the like, but there is little evaluation of those claims. These claims should be even more surprising than they would be for a horse because, whereas we do not know exactly what computational processes occur in horse, dog, or human brains, we do know exactly what computational processes occur in large language models. They are large, but conceptual simple, stimulus-response models. They predict the next word in a sequence from the preceding context. To be sure, learning to predict the next word is more sophisticated than watching for changes in your questioner’s posture, but there is still a great gulf between betting on word patterns and being able to analyze complex legal arguments, for example.
The difference can be encapsulated as the difference between knowing how and knowing that. Large language models know some word patterns (by “know,” I simply mean that they have the information). Sometimes, these word patterns match the desired output of an analysis. They know that in the sense that they know the pattern. When we attribute cognitive capabilities to these models, however, we are asserting that they know how to apply a skill. The bartender in the joke thought that the dog knew that the answer to each question was like a bark, but when the man and dog left the bar, the dog revealed in the punchline that it knew how to analyze baseball players (or that it at least knew more than just how to bark on cue).
In the world of GPT models, an analyzing skill is different from a word-predicting capability. Both involve some level of intelligence, but one should not confuse one skill knowing that words fit together for the deeper ability of how to reason and so on. So far as I am aware, there is no theory or even hypothesis for how a machine that learns word patterns could transform itself into a machine that has cognitive skills. There is no mechanism by which it can adduce deeper principles than just the language patterns.
Large language models can report the outcome of previously conducted (by humans) analyses by producing word sequences similar to those previous analyses in similar contexts. Given the number of parameters used to represent those word patterns and the fact that practically everything ever published on the World Wide Web is included in its training set, the parsimonious explanation for any of these models’ apparent skills is that they are simply paraphrasing previously seen patterns. Despite the large and growing (from version to version) number of parameters in these models, there are still many fewer parameters than potential word patterns. Multiple word patterns must, therefore, share the same parameters. Large language models are not just parrots, restricted to copying what they have read, but stochastic parrots, paraphrasing the text on which they were trained.
In the absence of critical analysis and in the absence of even a hypothesis of how a language model could attain cognitive skills, it seems much more reasonable to assume that any apparent incidence of these skills in a large language model is a manifestation of word probability patterns, nothing more. Words that are sometimes used to explain the appearance of cognitive skills, such as emergence, are a labeling of the presumed phenomenon. We imagine that with enough examples and enough parameters somehow or other we will observe deep cognitive skills where we know that the computational mechanisms for those skills are not contained in the model.
Recently, a petition has been circulating calling for a temporary moratorium on the development of more powerful language models than those that currently exist. According to this petition “AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one – not even their creators – can understand, predict, or reliably control.” “Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?”
The objective of this petition is based on a profound misunderstanding, bordering on hysteria, of just what the current and foreseeable state of artificial intelligence is. Large language models are not out of control, they are not digital minds, and they are not ineffable (they can be understood). They are, as I noted earlier, conceptually simple models that stochastically select words conditional on the context. There is zero chance that these models will automate away all jobs, or replace us.
Policies and regulations based on inadequate understanding of a situation will be ineffective, or worse. The fears expressed in this petition are based on such a profound misunderstanding of the technology that any resulting policies will be laughable, meaningless, or even harmful. There might someday be digital minds, but stochastic parrots are not among them. Casting the models as monsters under the bed and the scientists and engineers that create them as computational Frankensteins is irresponsible. The fear-mongering exacerbates the currently widespread mistrust of scientists and other experts, and may endanger individual lives. Today’s models are tools, they are not Skynet looking to eradicate life on Earth. Like any tools, they may cause harm when misused. If there are to be regulations, these regulations should be directed at mitigating current potential harms, not at the fear that these models are somehow inherently malevolent. We need to concern ourselves about how people may misuse models; we need not worry that the models will become our computational overlords.