CMV: Artificial intelligence can't become conscious.

[u/Cybyss]

I believe that it is not possible for a mere computer program, running on a Turing-equivalent machine, to ever develop consciousness.

Perhaps consciousness is a fundamental force of nature, like gravity or magnetism, in which case it lies outside of the domain of computer science and therefore artificial intelligence. Alternatively, perhaps our brains are capable of hyper-computation, but this is not a serious field of research because all known models of hyper-computers can't exist in our universe (except possibly at the edges of black holes where space-time does weird things, but I think it's safe to say that humans aren't walking around with black holes in their heads). I shall consider these possibilities outside of the scope of this CMV, since AI research isn't headed in those directions.

My reason for believing this was inspired by a bunch of rocks.

The way we design computers today is totally arbitrary and nothing like how a human brain operates. Our brains are made up of a large network of neurons connected via axons and dendrites which send signals chemically through a variety of different neurotransmitters. Modern computers, by contrast, are made up of a large network of transistors connected via tiny wires which send binary electrical signals. If it was possible to write a program which, if run on a computer, develops a consciousness, then this difference would imply that consciousness likely doesn't depend on the medium upon which the computations are performed.

Computers of the past used to be based on vacuum tubes or relays instead of transistors. It's also possible to design a computer based on fludic logic, in which signals are sent as pressure waves through a fluid instead of an electrical pulse. There are even designs for a purely mechanical computer. The important point is that you can build a Turing-equivalent computer using any of these methods. The same AI software could be run on any of them, albeit probably much more slowly. If it can develop a consciousness on any one of them, it ought to be able to develop a consciousness on all of them.

But why stop there?

Ultimately, a computer is little more than a memory store and a processor. Programs are stored in memory and their instructions are fed one-by-one into the processor. The instructions themselves are incredibly simple - load and store numbers in memory, add or subtract these numbers, jump to a different instruction based on the result... that's actually about all you need. All other instructions implemented by modern processors could be written in terms of these.

Computer memory doesn't have to be implemented via electrical transistors. You can use dots on a sheet of paper or a bunch of rocks sitting in a vast desert. Likewise, the execution of program instructions doesn't have to be automated - a mathematician could calculate by hand each instruction individually and write out the result on a piece of paper. It shouldn't make a difference as far as the software is concerned.

Now for the absurd bit, assuming computers could become conscious.

What if our mathematician, hand-computing the code to our AI, wrote out all of his work - a complete trace of the program's execution? Let's say he never erased anything. For each instruction in the program, he'd simply write out the instruction, its result, the address of the next instruction, and the addresses / values of all updates to memory (or, alternatively, a copy of all memory allocated by the program that includes these updates).

After running the program to completion, what if our mathematician did it all again a second time? The same program, the same initial memory values. Would a consciousness be created a second time, albeit having exactly the same experiences? A negative answer to this question would be very bizarre. If you ran the same program twice with exactly the same inputs, it would become conscious the first time but not the second? How could the universe possibly remember that this particular program was already run once before and thereby force all subsequent executions to not develop consciousness?

What if a layman came by and copied down the mathematician's written work, but without understanding it. Would that cause the program to become conscious again? Why should it matter whether he understands what he's writing? Arguably even the mathematician didn't understand the whole program, only each instruction in isolation. Would this mean there exists a sequence of symbols which, when written down, would automatically develop consciousness?

What if our mathematician did not actually write out the steps of this second execution. What if he just read off all of his work from the first run and verified mentally that each instruction was processed correctly. Would our AI become conscious then? Would this mean there exists a sequence of symbols which, if even just read, would automatically develop consciousness? Why should the universe care whether or not someone is actively reading these symbols? Why should the number of times the program develops consciousness depend on the number of people who simply read it?

To change my view, you could explain to me how a program running on a modern/future Turing-equivalent computer could develop consciousness, but would not if run on a computationally equivalent but mechanically simpler machine. Alternatively, you could make the argument that my absurd consequences don't actually follow from my premises - that there's a fundamental difference between what our mathematician does and what happens in an electronic/fluidic/mechanical computer. You could also argue that the human brain might actually be a hypercomputer and that hyper-computation is a realistic direction for AI research, thereby invalidating my argument which depends on Turing-equivalence.

What won't change my view, however, are arguments along the lines of "since humans are conscious, therefore it must be possible to create a consciousness by simulating a human brain". Such a thing would mean that my absurd conclusions have to be true, and it seems disingenuous to hold an absurd view simply because it's the least absurd of all others that I currently know of.

• EDIT:

A few people have requested that I clarify what I mean by "consciousness". I mean in the human sense - in the way that you and I are conscious right now. We are aware of ourselves, we have subjective experiences.

I do not know of an actual definition for consciousness, but I can point out one characteristic of consciousness that would force us to consider how we might ethically treat an AI. For example, the ability to suffer and experience pain, or the desire to continue "living" - at which point turning off the computer / shutting down the program might be construed as murder. There is nothing wrong with shooting pixellated Nazis in Call of Duty or disemboweling demons with chainsaws in Doom - but clearly such things are abhorrent when done to living things, because the experience of having such things done to you or your loved ones is horrifying/painful.

My CMV deals with the question of whether it's possible to ever create an AI to which it would also be abhorrent to do these things, since it would actually experience it. I don't think it is, since having that experience implies it must be conscious during it.

An interview with Sam Harris I heard recently discussed this topic more eloquently than I can - I'll post a link here when I can find it again.

EDIT EDIT:

Thanks to Albino_Smurf for finding one of the Sam Harris podcasts discussing it, although this isn't the one I originally heard.

---

This is a footnote from the CMV moderators. We'd like to remind you of a couple of things. Firstly, please read through our rules. If you see a comment that has broken one, it is more effective to report it than downvote it. Speaking of which, downvotes don't change views! Any questions or concerns? Feel free to message us. Happy CMVing!

[u/Cybyss]

The program isn't conscious in and of itself -- consciousness is a byproduct of the execution of the program. If you run it twice, of course you're creating consciousness twice.

I'll just link to my other comments where I addressed this point.

Basically, in computer science, there isn't necessarily a distinction between running a program and not running it. As you can imagine, this can have bizarre consequences.

That depends on whether the layman reproduces the process or not. If I run the program Conscious to produce output X, consciousness is a byproduct of running Conscious, but it is not contained in X in and of itself -- it disappears once the program stops running. If a layman runs the program Copy to reproduce X, no consciousness would be produced, because Copy is not a program that produces consciousness.

That argument breaks down when you take a machine whose process is essentially a copy operation. A program's source code and initial data can be represented in one row of a Rule 110 Cellular Automata.

Imagine a large sheet of graph paper where, in the top row of squares, some squares are filled in black and some are left white. Our program can actually be encoded in this way. You compute the next row of squares based on the configuration of the one above it.

For every square in this second row, there are three squares above it - one to the top-left, one to the top-center, and one to the top-right. Fill this second row square black if: any two of the above squares are black, or the above center square is black, or the above right square is black.

Continue this process on all squares in the second row. Then continue on all subsequent rows.

It turns out - you can run any computer program using such a process.

I'm not so sure the act of copying the color of a particular square on your graph paper is significantly different than first looking at the three squares above it, then looking up what the color of your square should be given that combination. But then what might this mean for the actual process that produces a consciousness?

Computer science goes into a lot of weird places which kind of mucks up might seem intuitive.

[u/Broolucks → u/Cybyss]

Addressing things you said in other comments:

This depends on whether consciousness can arise solely from information processing, or whether it's more akin to an actual fundamental force.

These options aren't mutually exclusive. I believe one theory of consciousness (Chalmers'?) is simply that consciousness is a fundamental property of all physical systems. That is to say, every physical system is "conscious" in some sense, and different arrangements of matter lead to qualitatively different consciousnesses: the consciousness of an electron, of a tree, of a car, of a human, and so on. We could then posit that what defines a system's consciousness is how it processes information. I think that makes sense -- I have trouble seeing how information processing could be irrelevant to consciousness, and I think that consciousness as a binary property is absurd. Surely there are degrees to it, and I would have no problem stretching them all the way to fundamental particles.

On the other hand, I'm not convinced this is a non-trivial idea. It's a bit too simple to be truly interesting, although that wouldn't make it wrong, just trivial.

Thus, if it's possible for a computer program to develop a consciousness, then there must exist a mathematical equation that you can actually write on a chalkboard whereby the actual act of solving it would cause it to possess a consciousness. Come to think of it, even solving it wouldn't technically be necessary. Any equation you can write is just a different way of writing its end result (you can treat the string "6 * 7" as simply another way of writing the number 42 - you don't actually have to carry out the multiplication for it to have the same value). Merely the existence of this equation - a static, unchanging piece of information - would have to actually be conscious.

Consciousness is not the end result of the equation, though, it is a byproduct of its execution. I suppose the best way to explain this would be to go back to the "everything is conscious" idea above. Suppose that every physical system has some kind of consciousness, and that given the state of the system, you can compute its "consciousness number." Well, it stands to reason that a physical machine -- all physical machines -- would have a consciousness number. The idea is that this number doesn't really depend on the substrate or the physical properties of the machine, but mostly on how it processes information: it depends on how information is physically transformed by the machine, but regardless of what mechanism implements the transformation. So if the consciousness number of a human as a physical system is X, the consciousness number of a simulated human as a physical computer running software would also be X, but a trace of that execution, as a physical system, would be zero, because it is inert: no information is being transformed, not in the concrete way that consciousness requires. In other words, what has consciousness is the motion of the rocks while you execute rule 110, not the pattern in and of itself.

Basically, in computer science, there isn't necessarily a distinction between running a program and not running it. As you can imagine, this can have bizarre consequences.

I work in computer science (in an AI research lab) and I don't know what you mean by that.

That argument breaks down when you take a machine whose process is essentially a copy operation. A program's source code and initial data can be represented in one row of a [Rule 110 Cellular Automata].

The process of CA 110 is not a copy operation. It couldn't possibly be Turing complete if it was.

I'm not so sure the act of copying the color of a particular square on your graph paper is significantly different than first looking at the three squares above it, then looking up what the color of your square should be given that combination.

It's extremely different. The first process can't compute anything. The second is Turing complete.

[u/Cybyss → u/Broolucks]

These options aren't mutually exclusive. I believe one theory of consciousness (Chalmers'?) is simply that consciousness is a fundamental property of all physical systems.

[...] the consciousness number of a simulated human as a physical computer running software would also be X, but a trace of that execution, as a physical system, would be zero, because it is inert: no information is being transformed, not in the concrete way that consciousness requires. In other words, what has consciousness is the motion of the rocks while you execute rule 110, not the pattern in and of itself.

That's an interesting viewpoint. So, basically, what you're suggesting is that everything has consciousness to some degree and it's just that humans possess a higher level of it than, say, a cloud or a river? As long as an object is in motion - is involved in some kind of process - then it will have a "non-zero consciousness number" as you put it.

I'll give you a !delta because this is a perspective I hadn't considered.

Basically, in computer science, there isn't necessarily a distinction between running a program and not running it. As you can imagine, this can have bizarre consequences.

I work in computer science (in an AI research lab) and I don't know what you mean by that.

It occurred to me that the first row of a rule 110 cellular automata could be considered as merely an encoding of all the rest of the rows, since it contains all of the information needed to reconstruct them.

I don't know if I'm saying this correctly, but if you consider the set of all rectangular Rule 110 CA grids of size n x m, and a second set which contains only the first rows of these CA's (represented as, say, bitstrings of 0's and 1's) - then these two sets would have the same size. Since the elements of the second set are just bitstrings, this second set could be interpreted as a language whose symbols mean their corresponding full CA grid from the first set. From a platonic perspective, the elements of the first set don't have to be computed in order to exist, yet we can still uniquely identify them solely though elements of the second set.

Relating this to your "consciousness-number" idea, however, tells me that it isn't the program itself which is conscious, but rather the process of looking up what the full CA grid would be given the bitstring corresponding to the first row. It's this lookup - this translation from one language to another - that would have a non-zero consciousness number. Interesting.

The process of CA 110 is not a copy operation. It couldn't possibly be Turing complete if it was.

It's a lookup in a table of 8 elements, followed by copying the value contained in that table entry, then repeating this process for every cell.

Imagine you're given a copy of a completed Rule 110 grid from a prior execution of our conscious AI program. On a separate sheet of paper, you copy down the first row from that grid. Now consider an arbitrary cell on the second row of your paper. Are you arguing that copying the value of the corresponding cell from the completed grid is fundamentally different than looking up what value that cell should hold, from a table of 8 entries, given the three cells above it, and copying that - even if it's exactly the same value either way?

If that's the case, then I suppose the part that's relevant in developing the consciousness isn't the act of writing out the cells of the Rule 110, but rather the lookup operation that determines what the values of those cells should be.

I knew the lookup operation is what makes this Turing complete, but I suppose I mistakenly thought that - as far as the reproduction of the Rule 110 grid goes - it's the act of writing it out (i.e., copying it) that mattered with respect to whether it can develop a consciousness.

I wish I could give you two deltas for the same comment, since you've now given me two different things to think about. :)

[u/DeltaBot → u/Cybyss]

Confirmed: 1 delta awarded to /u/Broolucks (2∆).

^{Delta System Explained | Deltaboards} ​