Reading: Douglas Hofstadter, “The Turing Test: A Coffeehouse Conversation,” in The Mind’s I, Chapter 5.
Hofstadter presents a conversation between three fictional friends: Chris, a physicist; Pat, a biologist; and Sandy, a philosopher. They disagree about whether computers could ever genuinely think, what kinds of similarities between humans and machines would matter, and how we should interpret the Turing Test. Sandy presses the case that mentality might depend on patterns of organization that could be realized in different materials; Chris and Pat raise challenges, including the thought that computers can only simulate thought, that living brains are unlike machines, and that computer programs are too rigid to have minds. The conversation doesn’t give us a decisive argument settling these disputes. It offers several different ways to understand what the dispute is about.
Last time we distinguished between a system’s computational capacities and whether it has mental states of its own. We also distinguished the idea that passing a demanding Turing Test guarantees intelligence from the idea that it provides good but defeasible evidence for intelligence. We’re now going to examine what else, beyond conversational performance, someone might reasonably want to know about a system. To do that, we need some basic ideas about dispositions, algorithms/software, and hardware.
Early in Hofstadter’s conversation, Chris and Pat challenge Sandy by comparing simulation of thinking to a simulation of a hurricane. A simulated hurricane may have simulated winds, waves, and rain, but nobody in the room where the computer is running gets wet. Why shouldn’t a program that perfectly simulates a thinker be just as lacking in genuine thought as the simulated hurricane is lacking in genuine rain?
That objection sounds forceful, but we should notice that simulating one kind of thing is not always the same as simulating another kind of thing. Suppose a computer program is set up to calculate sums for arbitrary pairs of numbers. When given 27 + 15, it returns 42 by executing an addition procedure. Is it merely pretending to add, or is it itself doing an addition? It seems perfectly natural to say that it really does calculate the sum. (A movie of someone adding numbers, or a program that merely plays back a recorded answer, would be a different case.) More generally, actually carrying out an arithmetic algorithm isn’t merely producing a theatrical imitation of arithmetic.
This gives us a philosophical question, rather than an answer: Is thinking more like raining, or more like calculating? If what matters to thought is having certain patterns of information processing, perhaps a system that implements those patterns really thinks. But if thought requires some particular physical process—as rain requires actual water, not merely bits representing water—then a computer could reproduce the relevant pattern in a simulation without thereby becoming a thinker.
Hofstadter’s characters also discuss how widely our concepts can extend. Hurricanes on Earth needn’t have exactly the same chemistry, temperatures, or surroundings as storms on distant planets. We can ask which features are essential to being a hurricane and which are merely features of familiar examples. Similarly, we shouldn’t assume that everything true of human thought has to be true of any possible thought. But we also shouldn’t assume in advance that the only features essential to thinking are abstract ones that can be reproduced in any material.
Notice that this debate is not settled by making a simulation more realistic to an observer. A remarkably realistic simulation might still not literally have the features being simulated. On the other hand, some activities—like calculating—can be performed in very different physical settings. We need to figure out which sorts of features genuine thinking depends on.
Recall our discussion of fragile performances on the Turing Test. A simple chatbot may seem intelligent for a few minutes, until someone asks an unexpected question. Another system might be able to respond appropriately across countless possible conversations, even ones nobody has actually tried. The difference leads us to an important notion: a disposition.
Something can have a disposition even when it never manifests that disposition. A glass vase may be fragile although nobody ever drops it. Being dropped or struck under suitable conditions would trigger the disposition; breaking would be its manifestation. The vase’s molecular structure helps explain why it has that disposition—what philosophers call its underlying or categorical basis. The vase doesn’t need to have broken already in order to be fragile.
We should distinguish three things here:
Consider two illustrations. Before he became Spider-Man, Peter Parker was a physically unimpressive schoolboy. After acquiring his superpowers, he often kept acting weak and clumsy to protect his secret identity. His classmates might see the same behavior as before, even though what he was capable of doing had changed drastically. Or imagine that someone secretly replaced your mother’s delicate crystal vase with an almost indestructible replica, while carefully staging events so that it still appeared fragile (without actually needing to break). What observers saw would be misleading about the vase’s real disposition.
The converse can happen too: a machine might give a few extraordinary answers by luck or trickery, even though its general dispositions are unimpressive. So the behavior we’ve actually observed is not necessarily an accurate guide to the full range of behavior a system is disposed to show. That is why short, gullible-judge versions of the Turing Test are inadequate even as tests of behavioral capacity.
Suppose, however, that we’re discussing a more ambitious hypothetical. A system doesn’t just appear to have the right dispositions; it genuinely would respond in the same sophisticated ways as a human across all the relevant possible circumstances. We can now ask whether this sameness of behavioral dispositions settles what mental states it has. A behaviorist will be inclined to say yes. Someone else might want to know more about the underlying basis of those dispositions. To make sense of that possibility, we need to understand what goes on inside computers.
An algorithm is a mechanical recipe for carrying out a task. To call it mechanical doesn’t mean that it has to be executed by a metal machine. It means that the instructions specify steps precisely enough that carrying them out doesn’t require the executor to invent a fresh method, exercise judgment about an unspecified issue, or already understand the answer the procedure is supposed to produce.
A familiar recipe may leave instructions like “cook until it looks right” or “add enough salt” to the cook’s judgment. A computational algorithm has to be more definite about what the next step is. Some algorithms are decision procedures: they are meant to answer yes-or-no questions. Others compute numbers, sort lists, search databases, or produce other results. You may hear philosophers and computer scientists talk about an effective procedure or effective algorithm. “Effective” here is a technical term; it doesn’t mean the ordinary notion of “gets the job done.” It means something more like “can be achieved by some explicit mechanical recipe using finite time and resources.” For us, it’s the general idea that matters, not the details or the label.
Consider a vending machine that accepts nickels and dimes and dispenses a Coke after receiving 15¢. We could describe part of its operation in a table:
| Current state | New input | Next state | Output |
|---|---|---|---|
| No credit | Nickel | 5¢ credit | Nothing |
| 5¢ credit | Nickel | 10¢ credit | Nothing |
| 10¢ credit | Nickel | No credit | Dispense Coke |
| No credit | Dime | 10¢ credit | Nothing |
| 5¢ credit | Dime | No credit | Dispense Coke |
| 10¢ credit | Dime | 5¢ credit | Dispense Coke |
The machine’s response to a nickel or dime depends on its current state. It doesn’t always respond to the input nickel with the same output. What happened previously matters. The table gives a simple state-transition procedure: depending on the current state and the input, enter a new state and perhaps produce an output.
This is a crude model of a computer, for several reasons. Our Coke machine has just one hardwired procedure—there’s no way to give it a new program and have it do something completely different. Also, it has a very limited, fixed collection of possible states. We haven’t distinguished its memory from its other components; our description of its state is essentially a description of everything the simple machine needs to keep track of.
More sophisticated computers have separate memory locations or registers that hold parts of the machine’s current state. One register might contain a number currently being worked on, another might record where a search should continue, and another might record which instruction is to be executed next. In a general-purpose computer, information in memory may even specify what program should be run. Procedures in more sophisticated computers can use conditional branching: if a test comes out one way, proceed to one instruction; otherwise, proceed to another. They can use loops and other complex patterns. A program can operate on stored data, and the computer can change the stored data or, under appropriate arrangements, the program itself.
These are significant practical advances over our simple Coke machine. Yet the same general idea remains: there is a system in a state, and rules specify or constrain what it does next given that state and any new inputs. In principle, if a physical computer has finitely many components and only finitely many distinguishable states, its operation could be represented by an unimaginably large table of the simple sort we used above. That would usually be a terrible way to understand or build it; the more sophisticated description exposes reusable structure that the giant table would conceal. But for some theoretical purposes, the table version is easier to work with.
A calculator or vending machine is ordinarily designed for a restricted class of tasks. A general-purpose computer can execute very different programs. Turing also helped develop the more ambitious theoretical idea of a universal computing machine: a machine that, given an appropriate description of another algorithm, can carry out that algorithm’s computation. In this theoretical sense, a universal machine isn’t restricted to a single fixed task or a predetermined list of useful applications.
Not every mathematical question has an effective procedure for answering it. What the notion of a universal computing machine means is that if a computation can be carried out by an effective algorithm, a universal machine could in principle carry it out when suitably programmed and supplied with adequate resources.
Turing’s idealized machines have unlimited/unbounded memory; real physical computers are only finite approximations to that ideal. A related distinction will matter throughout these notes: an algorithm is not identical with the particular physical machine that performs it. The instructions for sorting a list can be written in a notebook, programmed on several different computers, or followed by a patient human with pencil and paper. These may be different implementations of the same algorithm. Conversely, the same computer can perform many different algorithms at different times.
When we talk about ordinary computers we naturally picture silicon chips, electronic memory, a screen, and a keyboard. But none of those specific materials is part of the general idea of implementing an algorithm. If a mechanical arrangement of gears and levers follows the right rules, it can compute. If a network of switches operated by flowing water implements the relevant discrete states and transitions, it can compute. We can imagine building computing devices out of wooden pieces, rubber bands, or LEGO. Some computation can also be implemented using biological molecules or cells.
One might object: if the specific materials don’t matter, couldn’t almost any physical object be described as implementing some computation? The point isn’t that every physical system automatically counts as a computer merely because it can be described mathematically. The point is that what makes a physical device implement a particular computation is how its states and transitions are organized, not whether it has the familiar appearance or materials of a laptop. There may be important differences in speed, reliability, energy demands, and practical construction. But two physically different machines may nonetheless carry out the same abstract algorithm.
This raises the possibility that human brains are computing systems too. Brains contain networks that process information, retain traces of earlier activity, respond differently depending on their states, and help produce behavior. If some of these processes implement algorithms, then the brain is carrying out computations. The fact that it uses biological rather than electronic components doesn’t disqualify it from being a computer.
But how far should we take this idea? Does everything the brain does consist in implementing algorithms? And if the brain is a computer, does that mean our thoughts and experiences are nothing more than computations? Neither conclusion follows just from recognizing that the brain carries out computations. It might also perform other kinds of physical processes that aren’t computational. And even if all of its activity could be described computationally, we’d still have the further philosophical question whether those computational processes are sufficient to constitute genuine mental states.
We should also be careful about levels of description. Individual neurons may be quite good at reacting to particular electrochemical conditions, but it doesn’t follow that you, the person whose brain contains those neurons, have conscious mastery of those operations. Likewise, electronic circuits may be very reliable at certain low-level arithmetic operations, but it doesn’t follow that the AI system they implement will consistently answer mathematical questions correctly. The behavior of components and the capacities of the organized whole aren’t interchangeable.
One of Hofstadter’s themes is that living cells and sophisticated computers both involve intricate organizations of relatively simple physical processes. From close up, each may look like a lot of mundane local activity. That observation doesn’t by itself establish that machines think; it challenges the quick inference that a system couldn’t think because its components are doing something physically mechanical.
The possibility of different physical implementations of the same computation brings us to a second distinction: two machines can use different computational procedures even when the input-output task they perform is identical.
This is a distinction I particularly want you to retain. It’s easy to recognize that two computers can be made of different hardware yet run the same program. It’s equally important to recognize that two programs can reliably give the same answers while following different algorithms.
Suppose the task is to take any finite list of distinct numbers and return the same numbers arranged from smallest to largest. Consider this input:
[40, 10, 30, 20]
One possible algorithm is selection sort. Look through the unsorted entries for the smallest; put that value into the first available position of the output; then look for the smallest of those still left, and repeat. Schematically, its progress might look like this:
Input: [40, 10, 30, 20]
Staged/Pending Output: [ ]
Smallest: 10 → output [10]
Next: 20 → output [10, 20]
Next: 30 → output [10, 20, 30]
Last: 40 → output [10, 20, 30, 40]A different algorithm is insertion sort. Take the input values in sequence. Keep a portion already sorted, and insert each new value into the appropriate position within that portion:
Input: [40, 10, 30, 20]
Staged/Pending Output: [ ]
First: 40 → output [40]
Next: 10 → output [10, 40]
Next: 30 → output [10, 30, 40]
Last: 20 → output [10, 20, 30, 40]These procedures aren’t merely written in different words. They go about the task differently. The selection procedure keeps looking for the smallest unprocessed number; the insertion procedure maintains a sorted portion and finds a place to insert the next number. They use different intermediate states and may need different kinds of scratch information. If we could look into their working memory as they operated, we would have evidence about which procedure they were using.
Still, for every input list in the specified domain, both return the same sorted list. This isn’t just an accidental agreement on [40, 10, 30, 20]. Their input-output function is the same across all the relevant counterfactual cases: give either algorithm the same permitted input, and you’ll get the same final answer.
That conclusion concerns the final lists they return, not every observable detail of the way they run. The procedures may take different amounts of time or expose different intermediate steps. One might object that if the algorithms take different amounts of time, or go through visibly different intermediate steps, their observable behavior isn’t really identical. If we expand what we’re counting as their observable outputs to include those details, their behavior may be distinguishable. The point is that sameness of a specified input-output relation doesn’t logically imply sameness of the procedure that produces it.
Suppose we found two conversational systems that always gave the same outward responses to the same conversational histories. Could the internal procedures responsible for those responses nevertheless be different? Our sorting examples show why the answer is yes, in principle. It’s possible for different internal routes to lead to the same specified outputs.
Imagine, for example, an immense lookup table that assigns an appropriate response to every possible conversation history within some suitably bounded range. The system looks up the history so far and prints the assigned or “canned” response. Contrast that with a system that represents the content of the questions, updates its stored beliefs and goals, draws inferences, and decides what to say on that basis. We can stipulate that they give the same outputs in the cases under discussion, although the procedures are quite different.
A complete lookup table covering conversations of unlimited length would require infinitely many entries. Even a table covering only conversations up to some fixed length would be impractically enormous.
You might object: Why take such an unrealistic device seriously? And in any case, this isn’t a good analogy for how contemporary language models actually work. Both points are fair. The lookup table isn’t meant to be a realistic proposal for how to build an AI, or a description of how today’s AI models operate. It’s a thought experiment meant to isolate a philosophical question: If two systems had exactly the same conversational dispositions, could differences in the internal procedures producing their responses matter to whether the systems genuinely understand?
The behaviorist we discussed last time will be inclined to answer “No”, provided we’ve really stipulated the same full range of relevant behavioral dispositions. Other theorists may say that the lookup-table system merely mimics understanding, while the alternative system does something more like thinking. Such theorists still owe us an explanation of which differences among internal procedures make a mental difference, and why.
This is a question about what constitutes mentality. We can separately ask what evidence, short of directly inspecting the internals, a long and demanding Turing Test gives us about those internals. That’s one reason the distinction between logical sufficiency and good but defeasible evidence remains important.
When people doubt that computers could ever think, they often have certain assumptions in mind about what computing systems must be like. These assumptions are worth examining individually rather than treating them as a single objection.
Of course a program’s behavior depends on rules and on the state of the system that executes them. But that doesn’t mean the programmer wrote down a separate instruction choosing the system’s response to every situation it will encounter. Designing a procedure for finding answers is different from already knowing and specifying all those answers.
Suppose I write a program that searches for a solution to a mathematics problem I cannot solve myself. If the program finds one, I needn’t have secretly known it and encoded it in advance. Or suppose a program is designed to explore millions of possible moves in a game. Its designers may understand the search procedure very well while having no idea which move the program will select in a particular difficult position.
There’s another possibility too: programmers may create a complex system whose detailed behavior or internal organization they don’t fully understand. An explanation of how to build or train a system isn’t necessarily an explanation that predicts each of its future responses. Even knowing every instruction is not the same as being able to foresee everything that will happen when those instructions interact with large stores of information and a complicated environment.
A useful connection with Turing’s other writings is his distinction between complete rules of conduct—a high-level script saying what to do in every conceivable situation—and lower-level laws of behavior that determine or constrain what happens. Human beings don’t ordinarily have the first of these. They don’t consult a comprehensive list of explicit rules for every choice they make. But that doesn’t establish that our behavior is exempt from physical laws. Similarly, a machine could lack a complete high-level script telling it how to respond to every new situation, even if its operation is governed by low-level laws. This comparison need not assume that all physical laws are deterministic. Whether those laws determine a unique outcome is a further issue; the point here is that being governed by physical laws and explicit advance scripting are not the same thing.
The 1952 Turing discussion already explores machines that learn. Newman describes a proposed machine that tries different chess moves. After discovering a successful move, it modifies an instruction so that on the next occasion it can go directly to the successful choice. The programmer didn’t have to know the solution beforehand. Turing also envisages machines that learn more general methods from experience, so that learning one thing makes it easier for them to learn others.
This example is primitive compared with contemporary machine learning, but it illustrates the general point. A machine might begin with a way of revising its stored information, rules, or numerical parameters, and then acquire capacities the designers didn’t explicitly write into it one by one. Some changes may result from deliberate teaching; others from trial and error or exposure to a large body of examples. A trained system’s behavior can depend on what happened during its training, not merely on a fixed list of prewritten responses.
Hofstadter’s conversation imagines that people might be impressed by increasingly sophisticated computers while continuing to dismiss what they’ve achieved as soon as an understandable mechanism is discovered. But explaining how something works doesn’t by itself establish that the achievement isn’t real. We wouldn’t normally deny that a human really learns once we discover some of the neural mechanisms of learning.
You might object: Even if a machine can learn or surprise its programmers, why should that establish that it genuinely thinks? It doesn’t establish that. These examples instead remove a proposed obstacle: the assumption that a computer cannot acquire unexpected new capacities unless a programmer first possessed and deliberately placed each specific achievement inside it.
Turing’s 1950 paper distinguishes errors of functioning from errors of conclusion. An error of functioning occurs when a machine isn’t operating as its design or program calls for: a damaged component, a corrupted memory location, or a bug that prevents the intended procedure from being carried out. An error of conclusion occurs when the system operates as it is supposed to operate but reaches an incorrect answer—for example, it says 19 when asked 8 + 9.
A calculator implementing exact addition is designed to get arithmetic sums right, so if all of its relevant components work properly, it won’t make mistakes on those sums. But not every program is a calculator, and not every task has a simple, foolproof solution procedure. Hofstadter’s dialogue discusses weather prediction. A computer might use perfectly accurate input data and run its forecasting program exactly as intended, yet produce a forecast that turns out wrong. The problem is difficult, the model and data may not capture everything relevant, and some predictions require fallible approximations. That’s an error of conclusion without any malfunction.
You might object that the chips executing the weather model still have to perform their elementary electronic or arithmetic operations correctly. Yes. But there’s no contradiction in a system whose components reliably perform their local operations nevertheless giving a mistaken higher-level answer. Your neurons must also perform their ordinary electrochemical tasks for you to think and calculate; that doesn’t make you infallible at arithmetic.
For similar reasons we shouldn’t assume that artificial thinkers would have to be perfectly rational, unemotional, or free from confusion. Their capacity for careful reasoning, their systematic mistakes, and any emotions they might genuinely possess would depend on their organization. Whether a program simulates anger or genuinely feels it remains a separate question; but nothing about computation itself guarantees flawlessly rational outward behavior.
Several different ideas tend to get conflated here. A result can be unpredictable to us without being random. A deterministic program can produce outputs that are prohibitively difficult to anticipate in detail. Other programs use pseudorandom number generators, whose sequences are fixed by a starting state but can be practically unpredictable without knowledge of that state. Computers can also obtain values from physical processes used as sources of randomness, rather than generating every result from a wholly deterministic internal rule.
So the claim that every computer’s behavior must have been individually selected in advance by its programmer is false. That’s different from asking whether its behavior is ultimately determined by physical laws, and different again from asking whether it can make free choices. Even if one adopted a deterministic theory of computing, one would still need an argument showing why being deterministic rules out having beliefs, desires, or understanding. Perhaps some of the same considerations would apply to humans.
Thinking about sources of randomness also helps us see that the boundary of the relevant system isn’t always obvious. If an AI uses an external random-number device, is that device part of the computer we’re evaluating, or part of its environment? What about its persistent memory, a storage drive, an operating system, firmware, attached sensors, or a tool it calls? The answer can depend on which system and which activity we’re describing. For some purposes we discuss the central processor alone; for others the entire interconnected arrangement matters.
A modern AI assistant may include a trained model, surrounding software that provides it with instructions, tools, and memory (sometimes called a harness), and external services it can call. We shouldn’t automatically attribute every capacity of that larger system to the model alone. Nor should we assume that the boundaries of the hardware or software package sold as one product settle the philosophical question of what system, if any, is having the thoughts. We’ll return to that issue when we discuss Einstein’s Brain.
Sometimes the objection to machine mentality is not that the machine uses the wrong materials, but that it was made by somebody. If its behavior results from intentions that its designers had, perhaps any apparent thoughts or goals it displays are only reflections of their thoughts and goals. We should separate several questions about this.
Recall our distinction from last time. An artwork may express anger because of the artist’s intentions, without itself feeling angry. But a student could genuinely feel anger even if her parents genetically engineered her to be disposed to feel it in particular situations. In the latter case, other people’s intentional design helped bring about a genuine mental state; it didn’t merely create an artistic representation of one.
So being intentionally produced doesn’t, by itself, establish that an apparent mental state is only borrowed from the producer. If an artificial system has the right other features for genuine mentality, why should the additional fact that someone wanted or arranged for those features to occur disqualify it? Conversely, the fact that something arose without a designer isn’t by itself evidence that it has thoughts. We still need to ask what kind of system it is and what is happening in it.
Imagine an extraordinarily improbable sequence of physical events that leaves gears, switches, or other components arranged so that their causal operations implement a sorting algorithm. Nobody intended this arrangement or knew that the algorithm existed. Once assembled, though, it accepts lists and reliably sorts them. Would it cease to count as a computer just because no one designed it?
If we’re using computer to mean a physical system that genuinely implements a mechanical procedure, the absence of a designer doesn’t seem decisive. What matters is that the system’s parts are actually organized to carry out the relevant transitions. That is a claim about what the system can do, not about how it came to have the organization that enables it to do so. The same organized pattern of state transitions might arise whether the components were deliberately assembled or came together without anyone planning it. A single lucky sequence of movements wouldn’t by itself make an object a useful computing system; it needs the relevant stable causal organization.
Conway’s Game of Life is a cellular automaton: a grid of cells in which each cell changes according to a few simple rules depending on its neighbors. Starting from suitable configurations, remarkably complex patterns can develop. In fact, suitably arranged patterns can implement universal computation.
Usually a person chooses the initial pattern or designs the simulated computer within the game. But imagine that some physical system operating under similar rules happened, through its history, to develop an arrangement implementing computation without anyone designing it. In principle, the resulting causal organization could implement an algorithm. The possibility is fantastically improbable in many ordinary settings, but probability is not the same thing as logical or physical impossibility.
A simulation of the Game of Life runs on a computer built for that purpose. The further hypothetical is that a physical system might itself exhibit the relevant Life-like state transitions, with computational patterns arising there without anyone deliberately arranging them. The point of this thought experiment is not that the Game of Life is conscious, or that every interesting pattern counts as thinking. It is to pry apart the ideas of computation, artificial construction, and deliberate programming. They don’t automatically go together.
Now imagine two physically and computationally identical systems. One was designed and assembled in a robotics laboratory. The other arose over a long period through a process of variation and natural selection. At this moment they have the same dispositions, internal organization, and materials. Could one have thoughts of its own while the other does not just because of their different histories?
Someone could defend an account on which evolutionary history or deliberate design makes a mental difference. But that is a substantive additional thesis, requiring an argument.
There is also a potential confusion in the word artificial itself, which can suggest that an engineered system must be merely an imitation of the real thing. But artificial can be used in two importantly different senses. Artificial vanilla (first sense) means an imitation or fake substitute for real vanilla (which comes from vanilla beans). Contrast that to artificial light coming from lamps rather than the sun (second sense). This is still genuine light. In this second sense, something’s being artificial is a matter of whether it was crafted or occurred naturally. It doesn’t imply the thing has to be fake or merely simulated. Artificial light might not come from the sun but it’s still genuine light. It might come from the sun: for example, if we put a bunch of sunlight-absorbing materials in the direct sun and then brought them into a dark basement where they glowed, some might say they’re emitting “stored sunlight”; but it would still be artificial light in the sense of coming from an apparatus that we engineered.
When we call intelligence artificial, we might mean only that it was engineered, like artificial light. The label needn’t imply that it’s not genuine intelligence but merely a convincing imitation.
That distinction matters here: the fact that one of our imagined systems was engineered and the other evolved tells us how they arose, not whether either one’s intelligence is genuine.
Natural selection itself can produce organisms whose behavior is organized without any foresight about particular future situations. If our own capacities arose in this way, why should a system’s having been deliberately engineered, rather than evolved, automatically prevent it from exercising genuine capacities of its own?
We can make a related science-fiction hypothesis: perhaps an alien species originally engineered some machines, but those machines have been modifying, reproducing, and evolving for thousands of generations. Would their distant artificial origin matter more than their present constitution and capacities?
We now have enough distinctions to compare some philosophical views about what would be required for an artificial system genuinely to think. The following theories disagree about what makes it true that something possesses a mental state. That’s not exactly the same as asking what evidence would make us reasonably believe that the state is present.
On the kind of philosophical behaviorism relevant here, having the right mental states consists in having the appropriate dispositions to behave in various ways. A creature’s being disposed to seek water when thirsty, report that water is nearby, avoid what it takes to be dangerous, and react appropriately to new information isn’t merely evidence of a further hidden state; the right pattern of such dispositions is what having the relevant mental states amounts to.
It would be too crude to say that a behaviorist counts a single apparently intelligent reply as constituting a thought. We need to consider the broad, interconnected range of dispositions, including ones never manifested. Still, if two systems have all the same relevant behavioral dispositions, this behaviorist has no basis for saying that one is genuinely intelligent and the other merely imitates intelligence because of differences in hidden hardware or software.
This is one way of defending the view that reliably passing a demanding Turing Test could be logically sufficient for the mental capacities it tests. It is not the only possible interpretation of Turing’s own position, and it doesn’t follow that every form of mentality must be judged equally well by conversational behavior. You could be a behaviorist about beliefs but not about pains or fears.
A functionalist agrees with the behaviorist that what a system does matters, but usually denies that external behavioral dispositions alone settle which mental states it has. What matters is its internal functional organization: how various intermediate internal states are produced by inputs and other internal states, how they interact with one another, and how they contribute to later states and behavior. We can think of them as saying: it’s not just the input–output relations (how a system is disposed to behave in different environments) that matters; it also matters what algorithms the system uses.
For example, suppose someone wants water and believes that water is in the kitchen. These states may combine to produce a plan to walk to the kitchen. If the person instead comes to believe that the kitchen tap is broken, that change in belief may alter the plan while the desire for water remains. The functionalist is interested in this network of causal relationships, not just the visible sequence of walking and drinking. A belief’s or desire’s functional role is the role it plays in the broader causal organization of the mind. That’s why the position is called functionalism.
As we saw from the two sorting algorithms, the same input-output function can be realized by different internal procedures. A functionalist may say that some such differences are crucial. A huge lookup table selecting “canned” responses may not have the relevant belief-like and desire-like states, even if it gives responses that are correct in a stipulated range of cases. Another system might genuinely update internal representations, reason with them, revise plans, and so forth. A functionalist will look for the right internal organization, rather than demand only that the outputs look right.
But there’s an important qualification: functionalists don’t have to require identical low-level code or identical algorithms to every detail. Consider a computer-science professor who assigns her class the task of writing a program in Python or JavaScript that implements a particular sorting algorithm. The professor won’t expect all successful submissions to contain exactly the same source code. Students may use different names for variables, different loop constructions, or different ways to store temporary values. The submissions can still implement the same algorithm in the sense relevant to the assignment. Several programs that are distinct in terms of lower-level algorithms might realize the same higher-level algorithms or functional organization relevant to a mental capacity.
So we shouldn’t identify functionalism with the simple slogan “the mind is the program your brain is running.” Functionalists don’t have to identify your mind with that specific program. Their claim is that the right causal organization, described at the appropriate level, is sufficient for the relevant mental states, even when realized in different physical materials. This would be more like a kind or set of programs than one specific program. Deciding which organization or kind of program is the right one is a substantial philosophical and scientific challenge.
One further disagreement concerns whether the materials and physical processes implementing a computation can matter to mentality in their own right. Some philosophers think that physical features of the implementation matter independently of the abstract algorithm. Two systems might execute the same algorithm but differ in whether they have the physical features needed to produce thoughts or experiences.
For instance, perhaps certain biological or electrochemical processes are necessary for consciousness. A digital simulation of those processes might reproduce their information-processing pattern without reproducing the relevant physical features. This brings us back to the hurricane analogy from Section 1. A computer simulation can reproduce patterns in a storm without producing actual rain or wind. In a similar way, someone might argue that a computational simulation of the brain could reproduce its information processing without reproducing the physical processes necessary for conscious experience. Someone holding this view needn’t claim that only a human brain could think: an octopus’s nervous system, for example, might also have the required physical properties. But it would be a mistake to assume that just any material capable of implementing the same computational description would suffice.
You can still be a materialist if you hold this view, rather than behaviorism or functionalism. Materialism tells us, roughly, that mentality depends on physical reality rather than on a distinct nonphysical mind or soul. It doesn’t by itself specify which physical properties matter. In particular, a materialist could reject the claim that the same computational organization guarantees the same mentality across different implementations.
To bring this disagreement into focus, imagine a person whose biological neurons are gradually replaced by synthetic devices. Each replacement is designed to make exactly the same causal contributions as the neuron it replaces: it responds to the same inputs and affects neighboring neurons in the same ways. The process continues until the person’s brain is entirely synthetic.
At the beginning she clearly has her ordinary mental life. A functionalist will be inclined to say that, insofar as the relevant organization and causal interactions remain preserved throughout the replacement process, her thinking and experience remain preserved too. If no changes are detectable in how she reasons, reports sensations, or responds to the world, when exactly should we say she lost some mental capacity? And why would that particular replacement be the decisive one?
A defender of hardware-dependent consciousness may dispute the thought experiment’s conclusion. The synthetic replacements might preserve the causal functions described at one level without preserving some other physical feature alleged to be essential to experience. The challenge to that defender is to explain what the missing feature is and why it matters. The challenge to the functionalist is to explain why preserving the relevant organization really is enough—not merely why there would be no behavioral evidence of any change.
The neuron-replacement case therefore sharpens the disagreement: what exactly must the replacements preserve? Is preserving the original neurons’ functional contributions sufficient, as the functionalist claims, or must the synthetic replacements also preserve some further physical property? The person’s unchanged outward behavior alone doesn’t decide the issue.
There are four contrasts I’d especially like you to be able to keep separate. They aren’t four names for the same difference.
First: the behavior we’ve observed versus the behavior a system is disposed to produce. Not everyone notices this contrast when first discussing the Turing Test. But once it is pointed out, most people agree that what really matters to judging a system’s intelligence isn’t a few lucky successes; it’s whether the system would reliably and flexibly succeed under a wide range of relevant conditions. Spider-Man’s apparent clumsiness and the secretly reinforced vase show why an observed performance can be misleading about underlying dispositions. This point isn’t the main divide between behaviorists and functionalists; both can agree that the dispositions matter more than a limited sample of observations.
Second: a system’s input-output dispositions versus the internal algorithms or functional organization that produces them. Here the philosophical disagreement becomes more substantial. A behaviorist thinks that the first, suitably rich set of behavioral dispositions is enough for the mental states we’re asking about. A functionalist thinks that the first is not enough by itself: the system must also have the right kind of internal causal organization. But the functionalist thinks that such organization is sufficient for genuine mentality, regardless of what particular materials implement it. Other philosophers may agree that internal organization matters while denying that it is sufficient on its own.
Third: an abstract computational or functional organization versus the physical system that implements it. We can construct the same kind of computation in different hardware. If someone thinks that, nevertheless, the physical properties of the implementation make an independent difference to whether mental states are genuinely present, that person rejects both pure behaviorism and the substrate-independent version of functionalism just described. Perhaps an appropriate organization implemented in some materials would support mentality but the same organization implemented in other materials would not. That is a further claim requiring argument.
Fourth: what a system is like now versus how it came to exist. Two systems might have the same present capacities, organization, and physical makeup while differing in whether they arose through intentional engineering, natural selection, or accident. If someone thinks that this difference in origins itself could make one genuinely intelligent and the other not, they too reject both behaviorism and functionalism as described above, since neither theory makes a separate requirement about how the system was produced. Of course, a system’s history might affect its mentality by changing its present organization. The stronger, controversial thesis is that its history matters even when those present features are held fixed.
Finally, remember the relationship with our earlier discussion of dualism and materialism. We can approach all these questions as asking:
A dualist cannot simply assume that computers are excluded from having mental states because they are machines or because other agents built them. That exclusion would need an argument about which physical systems can be associated with minds or souls. Conversely, someone who favors functionalism or a broadly materialist view still needs reasons for their proposed account of what constitutes mentality.
Our next reading, by Melanie Mitchell, investigates a related but distinct question: When we see an AI system performing impressively, how can we tell what capacities and mechanisms really explain its performance? Those are questions about evidence and scientific investigation. Even if we answer them very well, the further philosophical dispute about what makes mental states genuine won’t automatically disappear.
These terms may be helpful when reading about present-day AI, but none of them should be treated as deciding whether a system has mental states of its own.
The distinction between a model and its harness is one contemporary instance of our more general question about which parts of a system are responsible for which capacities. Later we’ll ask an even harder question: if a program can support mentality, is the bearer of that mentality the hardware, the abstract program, or a particular running process?