The Chinese Room argument, proposed by John Searle, highlights a fundamental philosophical challenge: while computational processes can simulate intelligent behavior, they may lack genuine understanding or consciousness because they lack the necessary causal structure that biological minds possess. This raises important questions about whether language models and other AI systems, despite their impressive capabilities, truly understand or merely simulate understanding. The debate touches on key distinctions between finite state automata (with fixed memory) and Turing-complete systems (with potentially infinite memory), and explores whether computational organization alone can produce genuine consciousness or if specific physical causal structures are required.
Chinese Room Argument & AI Consciousness | Philosophy of Mind Explained
Added:You know, a mathematical model of a black hole is not a black hole.
A weather simulation is not weather. Okay?
A computational process simulating the human mind is not the human mind.
There's this gap. There's this gap between our abstractions, between our computations, between our math and physical realization. And it's fascinating.
Welcome back to Tim and Keith's Philosophical Steakhouse.
Exactly. Our fireside chats. I think it's been almost a month since we did the one that or maybe five weeks even that did that did the one that led to the, uh, uh, you know, this this like, doomed debate analysis which led to me going on doom debates and whatever. Right.
So that was pretty funny. Yeah. So, um, it was your fault that I didn't get to bed until about 3:00 in the morning the other day, because I wanted to watch the Dune debate, and I thought you did a very good job.
I mean, you know, obviously the comments were really positive.
I think everyone went from hating Keith Dugard to thinking that Keith was intellectually honest and not such a bad guy after all.
So no, I thought it was really good. And props to you.
Listen, I, uh, I'm very impressed with how you handled yourself.
The whole situation was great. Unfortunately, I did get the impression that both of you left feeling that you'd won the debate.
What do you think? Well, um, so I wasn't.
I mean, I actually wasn't in it to to win that that debate.
It's more like I'm always. And nobody ever believes me.
But but my goal is always the pursuit of, you know, knowledge. Right.
And and or sharing and and whatnot. So I think, um, I think we, I think we made a lot of progress in the sense that, um, he agrees and, and hopefully most people agree. Now, you know, there's, there's a technical sense, like there's a very technical sense in which, um, LMS and again, like, you got to be careful about how you define that, you know, and whatever. Not LMS systems not not oh one not but I mean, just an LM, you know, sort of, uh, auto autoregressive fixed context window type thing. I think there's a sense in which people understand that's not Turing complete, like in this, in this technical sense. Right. Um, there's still these open questions or folks aren't that aren't as convinced as I am.
Let's say they're not as convinced as I am that that actually has practical consequences. Like, I think you're in that same that same camp, right. Which is that. Okay, fine.
There's this technical point in which in which that's correct.
However, practically speaking, you know, LMS can, uh, generate, generate Python programs that you could run in a system if you wanted to.
They do have like a very large, uh, context window.
And so if they're trained, right, they can do a lot of important things. And so I understand that that's still an open question. And I have more work to do to convince folks that there is a practical consequence.
Um, so that's fine. Yeah. Like, I mean, I'm happy with that.
It's certainly true that if you look at the statistical distribution of programs on GitHub, it will have a power law distribution, which is to say most of the programs will be short, low in complexity.
And you distinguished the Linux operating system as being a really complex program. And any complex program could be decomposed into lots of smaller programs.
But we should also distinguish the difference between something like a Python program and a Turing machine program.
Now there's this famous programming language called Brainfuck.
And that actually, are we allowed to say that?
Are we actually are we allowed to say that on YouTube, I wonder I think as long as you don't use the F word within the first seven seconds, then you don't get dinged for advertising.
Oh, okay. Okay. All right. Yeah. And because machine learning models don't really understand, they won't know that we used it legitimately.
Oh, I got you, I got you. Yeah, that's that's a good point.
Yeah. Yeah. We didn't use it in an offensive way, but, um, no, there is there is a difference between something like a Python program and a Turing machine. So a Turing machine is this hypothetical physical machine that can, you know, it's got a it's got a program. And of course, the program is fixed because when we talk about language models being able to accumulate information in their context, the analogy breaks a little bit with the Turing machine and a Turing machine. It's a finite state automata, and it's fixed. And then it can write to an infinite memory on, on both sides to a potentially.
Yeah, to an unbounded, potentially infinite memory, not an actually infinite memory, folks. Like nobody nobody's saying that that exists like. No. You're right. I mean, what distinguish the difference between potentially infinite and infinite?
Well, so potentially infinite is, is a, um, let's say a procedure that can that can keep going in principle for, you know, forever. But it may not ever. I mean, you may never get to the end, right? So for example, time like the flow of time in the universe, um, may or may not, you know, it may be potentially infinite, like the universe will just keep going, uh, on and on and on, but it will never reach the end, right?
There will never be an actually infinite, um, time span.
And so that's the difference. Like, for example, if somebody were to tell you that, um, that they had a register in a computer that was, uh, arbitrary precision, you know, then what that would mean is that this thing could as necessary, Expand to to have more digits.
More digits of precision. Whereas if they told, you know, I have a register that's, you know, actually infinite.
That's a very different kind of thing.
Like that's, that's a thing that in a finite amount of time can do an infinite amount of work. Right. Versus saying that this is something that over an infinite amount of time, it can do an infinite amount of work.
So that's really kind of kind of the point is, do you like in a single time slice, do you have an actual infinity, you know, sitting there? Um, and we don't know how to build that. Like we don't know how to build an actually infinite, you know, register or but we do know how to build things that we can keep adding to and adding to.
So I felt that one confusion, uh, coming from earlier on was the difference between the neural network weights being a finite state automaton and the input being the finite state automaton.
Would you agree with that? Yeah. I mean, there's this and there's this really subtle distinction because as I've said so many times, and I hate saying it every time I say it, because it doesn't do justice to, uh, to Turing. And is that a Turing machine is just it's nothing but a finite state automata with an unbounded read write tape. And so you have to think of that unbounded read write tape as like an extension really of its state space.
Right. Because it can go and modify it and read it again. So it's basically, you know, it's it can grow its state space to as large as necessary to solve the problem.
Um, and that's why, uh, that's why it can explore, you know, an unbounded set of computations, set of states, like whatever, because it can keep expanding, right? Um, and I think there's, there's all.
And it's become more apparent to me over these last months. Right.
Debating this issue is that there's always been almost like this kind of ambiguity or sort of unfortunate choice of language and computer science where where they refer to machines as both a model for a physical thing that you can build, as well as a notional mathematical model that maps to that as well as a class of algorithms. Right.
And so that leads to some confusion, and I actually hope to clear that up in the future, or to present it in a way that makes more sense.
Um, but again, I think people have agreed. Okay.
There's a technical sense in which this is right, but does it make any practical difference? Right. And I would 100% admit I haven't communicated enough or clearly enough.
Um, my view on why it practically matters. And I could be wrong.
You know, I don't I don't as yet have I have strong opinions about it.
But, um, at the same time, it's it's not as clear cut as you know, it's not as clear cut whether or not an LLM based system, an LLM plus some Turing machines, plus some, you know, hash table, it can read, write to or whatever the capabilities of that system are much less clear, you know. Okay. But another source of confusion is, you know, Leon asked you about the human brain, and he said, it's a it's a finite machine. And you said, yes, you accepted that the brain is a finite state automaton, but it runs an algorithm which is a Turing machine controller. So we can effectively expand our memory, maybe even internally. But let's just use the the external version of that. So then you're getting into this kind of quite ambiguous zone where you're saying in the class of Turing machine, sorry, in the class of finite state automaton algorithms, there are one kind which can expand its memory.
And then there's another kind which can't.
And you're saying the types of algorithms which are learnable on, let's say, a language model, a bog standard language model, are not the kinds that can expand their memory. Yeah.
And so here's this ambiguity I was talking about, because in computer science you would not call an algorithm.
So if you have an algorithm which is a pushdown automata algorithm or a Turing complete algorithm, you would never call it a finite state automata algorithm, even though the controllers for all of those machines are a finite state automata. Right?
So, so it's this funny, you know, it's this funny ambiguity where like, yeah, at the end of the day, all these machines are a controller, which is a finite state automata having access to, you know, no memory in the case of a finite state automata or a stack or a, you know, read, write tape or whatever. Right.
And so, yeah, there there's an ambiguity there.
And what I'm saying is, in the space of all algorithms, um, there are algorithms that utilize external memory, and there are those that don't and that currently when we train Llms, they are trained in an environment where, um, they don't have potentially infinite read write memory. They oftentimes have potentially infinite write right memory like output.
Right, but not read write memory. Okay, I got that.
So what you're trying to do is imagine there's a diagram and you have finite state automata and you have Turing machines.
And you're saying that rather than partitioning the space of algorithms neatly between those, clearly we can take an example of a Turing machine algorithm because a Turing machine has this codebook, which is the finite state automata. And you're saying in the context the transition function. Yep yep yep. In the context of a Turing machine, we call that algorithm a Turing machine algorithm.
But all of the algorithms that run on finite state automata.
So on basic language models, you call those a finite state automata. Yeah.
I mean, that's that's my claim, right?
My claim is that they're that they're the algorithms that we end up with when we train, um, these machines where they're set up to run as finite state automata, we end up training finite state automata, you know, algorithms. Right. And I think and as I've said, I think that there's there's more work I have to do there to convince people of that fact. Right. Which is that it isn't just it's not just that they're in construction finite state automata because they have these fixed windows, right? They have these fixed context windows. Um, but what I'm saying is that in addition, in my opinion, the algorithms that we access when we train those things come from the space of finite state automata algorithms rather than from the state of pushdown automata.
You know, um, uh, Turing machines, etc.. Um, okay.
And that's, that's part of the extra work I have to do. Right? Okay.
But to be clear, you're saying that the substrate so a native language model can in principle, it's a similar thing to you against Schmidhuber was saying in principle, an RNN can be trained to be a Turing machine, but you can't train it with stochastic gradient descent.
And he wasn't just talking about the infinite precision thing.
I think he was talking about some other examples for solving parity, where they can be something more than what they are if they are trained with stochastic gradient descent. Because the thing that confuses me is look at how ChatGPT is trained now. It's trained to do Rag natively.
In fact, I interviewed Patrick Lewis. He's the guy who coined the term rag, you know, a deep expert in rag. And he was telling me that's how they do it at cohere. When they train their models, it goes into a special rag mode. So it kind of tightens up because there's loads of things you need to optimize when you're training Rag and they are training these models on mass to expand their memory, you know, through an external store. So and when I say expand their memory, let me be clear. These things are not expanding their physical memory, but they are bringing into their context things from the outside, and they are potentially putting things out of their context to the outside. Now, Rag is a special case because I think it's only bringing stuff in. It's typically not bringing stuff out, although there's no reason why it couldn't do that in principle.
So I'm asking you the question how do you know that training models in this way wouldn't allow them to get into that space of algorithms, which you would call Turing complete? Okay, so and I'm going to kind of skip by because I know we don't want to get bogged down like in the whole the whole debate. So I'll try to skip past some things and just say, look, there's a lot of work that needs to be done to kind of clean up the language here, you know?
So I'll just say this, which is, um, as far as I know, the way in which rag is trained never has a closed loop like it doesn't.
Both write data to the database and read data that it's written to the database. It may do a retrieval of some data, you know, internet search or whatever, and kind of, um, run through that, write auto autoregressively sort of just like pass through that as part of its context and build up some hidden state and like do something, but it doesn't close that loop.
So it doesn't both read and write to the same unbounded store.
But as I've said, like so many times, if you if you alter your architecture to where you do have that, like you do have this closed loop of the ability to read and write to an unbounded store, then you have a Turing machine, and then you have the potential of training or finding algorithms, let's say, from that larger space of Turing machine algorithms. But as far as I know, like anybody who's attempted to train such systems, fails.
And I mean, because you run into like really basic problems.
So, for example, let's suppose I'm trying to train such a machine. All right. Like I've set it up, it can read and write to a database. And I start and I start running the algorithm. Now in order to do training right, I have to run it until it gives me an answer.
And if I want it to be a Turing machine, I need to allow for it to run for some unbounded amount of time, some unknown amount of time. Right? Like it has to generate a stop token.
It has to say I'm done. Now here's the answer.
Okay, so I go to train it. I start running the thing and you know I'm waiting and it doesn't stop. And I'm like, all right, it's been five minutes here and it hasn't completed yet on this, um, this sample, this batch. Right. Like, what do I do?
Do I let it keep running or do I prematurely terminate it and throw that run out? What do I do? So this is the problem is that Turing algorithms can run for an unknown amount of time. They may never halt.
So you're immediately faced with a very difficult problem during training, right? Yeah, I'm glad you said that.
So the problem is during training. So you could. Yes.
The you know, the whole stopping token thing, that's an artifact of our chef. So we teach them to stop.
We teach them to prefer a certain verbosity and length and so on.
Certainly when you just train language models just on the raw distribution of text, you can just you can just run them forever.
And they never say stop. Well, I mean, in order for it halting, stopping is a basic a basic requirement for, for algorithms.
So for a Turing machine to, to to do what's called decide a problem, um, it has to halt and give an answer. It has to halt and either say yes or no. Right. And so there are many mechanisms by which you can indicate that the computation is done.
You can have a special you can have special states that you enter. Right.
Like if I enter the halting state or halting state six or whatever, it doesn't matter. Like then that I'm done.
Or you can write some special thing to the tape.
You can, you can say like, you know, here's the answer or the answer is between A and B, you know, open paren, close paren, whatever.
But you got to have some way of indicating that the computation is is done somehow. Um, and, and here's the problem during training is exactly what I said.
How like, if it hasn't halted yet, what do I do? Yeah. Okay.
But I think we're kind of conflating two things.
So on the one hand, we could talk about today's language models that are trained on internet data. And we can hypothetically imagine that there's some kind of transfer between that manifold and the manifold of Turing machine programs. Another thing we might do is let's just throw it all away and start again. Right?
So why don't we create a massive data set of Turing machine algorithms.
Some of them will halt and some of them will not.
Now, I believe there's some interesting structure of Turing machine algorithms. It's fascinating to think about what that structure is. And what we might do is just heuristically run a whole bunch of those sampled programs because we could build a generator, or maybe we can be a bit smarter about it.
We can create these little units of Turing machine components that crop up all over the place, because I think that there is it's like Francois Chollet said with the kaleidoscope hypothesis, right, that there's all of these priors of the universe, and then they additively compose together to produce the kaleidoscope and intelligences, reversing back from that kaleidoscope into the priors which went into it. There must be something similar in Turing space, and this is related to Chomsky's ideas about nativism and so on. There's there's probably like a nice set of primitives that could compose together in an interesting way, and we could generate a bunch of Turing machines based on those primitives, and some of them would halt and some of them wouldn't.
And there'd be a very interesting structure there.
And maybe we already run them. So we know if they halt or not, and we build a large language model on it, and we get this manifold of Turing machine programs. And then maybe we try and map that manifold to the manifold that already exists.
But that, I think, would be an interesting thing to do.
I mean, I agree it would be it would be very interesting.
Um, and this is where this is where this falls into the category of what I've been saying for a long time, which is we don't yet know how to train, um, from a very rich space of Turing complete algorithms.
And so maybe the proposal you just made will be a pathway towards that, right? Um, maybe that's not even. Maybe we don't even need to do that much human human engineering. You know, maybe if somebody just finds some clever tricks, some some improvements to SGD or some alternate, you know, hybrids of that plus neural evolution or something like that, they'll, they'll find ways to search the space. Right. And I totally believe that that's possible. I'm just saying empirically so far, and it's kind of an empirical question empirically so far.
You know, people have failed. And it's a very fascinating and, you know, interesting result. Like, why why is it so hard?
Um, there may actually be some deeper truths there.
And I don't know, it's it is a fascinating question.
The space of Turing machine programs, of course, is is infinite, right.
Or maybe you would say potentially infinite. Yeah.
Countably. Countably infinite. Countably infinite. Yeah, yeah.
Because I guess the broader problem in AI is overcoming ambiguity and searching, because it seems that we wouldn't need to sample all of those Turing machine programs to come up with some statistically salient algorithm. But maybe that's not the case.
What do you think? I think it's a big mystery.
I think it's a big mystery. So. And here's where I may be, like, woefully out of date with, uh, computer science knowledge.
So if I am, I apologize. And actually, if anybody out there can can educate me on on this, I would I would love to know more, which is as far as I know, the last time, the last time I looked into this, we know so very little about this space of, um, of algorithms, right. Of the Turing. Turing complete algorithms.
We just don't know a lot about them. I think there's sort of a few, a few facts that are known, like, you know, most, most algorithms, um, never halt or or, you know, or this, this type of thing, we know that it's countably infinite, um, that type of stuff.
But I think in terms of the structure of the space of all algorithms, um, either a lot, a lot isn't known or I'm not up to date on the lot that is known. But I think it's it's fascinating, I think there could be lots of interesting structure there, especially if like you say, um, you know, you mentioned this, this, this language earlier, but I'm just going to call it, uh, the BF language. You folks can spell it out if they want to. Uh, and it's, it's like the simplest kind of mapping from a Turing machine to a programming language.
You know, it's got like characters for move left, move right, right.
Something kind of open a block, close a block, and like, that's about it, you know, not not much more than that.
And that's why it's called BF. Because if you ever tried to write a program in it, you'd you'd want to quit really soon and go go find something else to do. Um, but maybe maybe.
And of course we have our own we have our higher level languages. Right?
But the curious thing about the vast majority of programming languages that we've developed is that, um, if you just generate strings in that space. Most of them are not valid programs, right? Like, you have to kind of follow some type of a grammar in order to generate valid programs.
And even if you follow that grammar, uh, the vast majority of programs are like probably pretty useless, like bugs.
Or they just like, don't do anything or they infinitely loop. You know what? If somebody could come up with a and I'd love that. If anything works being done on this, please, please let me know. But what if somebody could come up with a language where almost every single program you generate is a useful program of some kind? Yeah, that would be cool.
A couple of things on that. Do you remember when we spoke with Daniel Roberts from MIT and I. Daniel, I'm sorry we haven't released your show yet. We we will as soon as possible.
Whenever there's a lag releasing a show, it's because of love and care on on my part. It's because I want to do a good job of it. But he was talking about quantum field theory. You are a bit of a perfectionist sometimes. I am is the curse of perfectionism, but we're not going to do that on on Keith and Dagger's Philosophical Steakhouse. We're just going to stick them out.
But no, he was saying, you know, in quantum field theory, they also have this ridiculous blow up of particle interactions and so on.
And the way they reduce the complexity are these kind of heuristics like sparsity and locality and, um, symmetry and whatnot.
And, you know, as you say, we could just enumerate all of the possible combinations of the space. Most of that wouldn't be valid Turing machine programs. But, you know, also there are many, many abstractions on top to write a generating function of useful Turing machine programs. But then I'm asking the question, well, we keep coming back to this on MLC, that there's no true Scotsman, that we need to go to the fundamental level of computation.
Wolfram, you know, talks about the ruliad and these kind of graph cellular automatons. We're talking about Turing machine space, but we also are guilty of talking about the same kind of thing with agency. Or you need to have the low level particle dynamics. You need to have the real ontological particles. There's no point having some higher abstraction, because if you look at what Sholeh is doing and folks advocating for discrete program search, they're kind of saying, well, forget about the Turing machines. Why don't we start with something that we believe to be fundamental? And those are spell keys, cognitive priors, this core knowledge. So if you start with the premise that there is core knowledge in the universe, maybe not the universe, but our minds, I'm not sure if that's a distinction which matters in this case, but if you start from some structure and you write a generating function over that structure, which is what Chollet advocates in the Arc challenge, would that be good enough?
I mean, this is there's so much I have so much, uh, that's so appealing to me. That approach is so appealing. Right.
And I think this is pretty similar to what Waleed Sabbagh, you know, tells us to write, which is that there are these, um, universal templates. And, and, you know, if we've got enough of those and if they're, they're well specified enough, um, then yeah, you can have these grammars that generate, um, super useful algorithms. So definitely I think there's a lot of appeal there. Um, I know that I know that it's criticized a lot. And people, you know, say, well, it hasn't worked or this project failed or that project failed.
Um, but I, I do kind of feel that those may be a very interesting path forward, um, at some point. And actually, I think, um, if I were to guess, if I were to guess where we'll end up some long time from now, like 50 years from now or more is we'll have hybrid systems. Right? So there'll be components that we've trained, constructed, generated, whatever it is through those mechanisms, there'll be components that are handcrafted by humans, and then there will be machine learned, um, components that we've learned how to, uh, put together an an intraoperative way. Right.
That'll probably really be the future of of, uh, artificial intelligence, if I had to guess. You know, like, the pendulum will swing eventually. People will like it will kind of plateau in what we can do with stochastic gradient descent and ever larger, massive, you know, training data centers.
And then, uh, you know, the sort of dreamcoder and program synthesis and art challenge and spellcheck and template type approaches will start to produce more and more, and then we'll figure out we still need a little bit of human ingenuity to make sure they're all working together.
Well, um, so yeah, I'm a big fan of those approaches. Yeah.
And it gets back to the functionalism discussion.
We'll touch back on this again when we talk about Searle.
But there's always this notion of the universe is hideously inefficient.
You know, it's created these Beautiful kind of recursive minds that we have, and we have all of the priors that help us understand the universe and how this happened. I mean, we can talk about the evolution of language and cognition and nativism and so on, but it's very, very mysterious. But we know it was a very kind of long, drawn out, inefficient process. And, you know, evolution had a lot to do with it. But then there's the question of, okay, well, now maybe we could design something which does the same thing, right. And that's kind of what functionalism is talking about. It's saying, if we could design a system which performed the same abstract functions as the thing that we did, and it has the same behavior, then it might be isomorphic.
Now, of course, someone like John Searle would say, well, there's an ontological difference. There's a difference in the the causal structure. Even actually in functionalism, there's huge debates about what kinds of causal structures you need to maintain. Like Chalmers, for example, he says that, you know, there are certain counterfactual trajectories.
And, you know, you need to have a very rich causal structure in order to kind of like, you know, create a good enough simulacrum of the real phenomenon. But the point is, though, that I, I think, I believe this, that there are probably uncountably many other configurations of systems which would be as if intelligent.
And it's, it's it's kind of like the no true Scotsman argument to kind of say, well, that one's not really intelligent or that one is intelligent, when in fact, in basically every situation they behave in the same way. Yeah. And, and this is kind of what happened in our discord. Right? Because, um, fresh off the fresh off the, the episode that we did on whether or not one could reason, um, which which led to all kinds of debate and, and abuse on YouTube and, you know, uh, and Leon's response and me going over there in parallel to that in our discord somehow. And I forget how it was.
I forget how what the origin was. We take it back and trace it.
We got into this huge you know, I don't want to call it a flame war, but this huge internal mlst discord debate about John Searle and the Chinese room argument, because I mentioned I mentioned the Chinese room argument. Right. Um, and one thing that happened like early on in that debate is, is people accusing slash mischaracterizing.
Searle is saying that he denies that a machine can understand.
And in fact, that's the total opposite of what he said.
You know, he's been saying this whole time, right?
That the only thing that can understand are machines, right?
And that we're a machine and the current and currently the only examples of things that can understand are biological machines.
Okay. And but that doesn't mean you couldn't build a different kind of machine like.
His only claim is that it has to be a machine. It has to be.
It has to have a realization. It has to have like I mean, I think the correct term too is embodiment, right?
Like it has to have a physical realized embodiment or not.
Maybe not. Let's just call it a realization.
We'll get into embodiment later, and that that realization has to have some sufficient causal structure. Right?
As you as you said, like an internal causal structure.
Um, he as far as I know, he doesn't exactly specify like what that internal causal structure is or should be or whatever.
He's just saying that it has to be there.
So you can't just have like an ethereal algorithm that's kind of running ethereal on some substrate that doesn't have that causal structure. You've got to have this realized causal structure. And I agree with that.
Um, and I also agree with you and Tim that there can be many different forms or many different substrates upon which we could build that, that causal structure. Um, and maybe and maybe there's more than one causal structure. It's like, okay, this causal structure is sufficient for understanding this alternative causal structure is sufficient for understanding.
There may be a whole bunch, right. And there may be a whole bunch of different physical ways in which you can realize that causal causal structure, but it does have to be realized.
So John Searle is quite possibly the most misunderstood person or philosopher on on the planet. I think the Chinese room argument is probably misunderstood by about 99% of people on online.
You've pointed out a few of the confusions.
You know, some people say that he is saying only biology as a machine substrate can give rise to consciousness and therefore understanding. And he doesn't say that as you said.
He says it's the only existence proof.
And I'm not a huge fan of the actual Chinese room argument itself.
I think it was a product of the time, and he was talking about semantics.
So the broader theme is he's talking about certain patterns of physical causation. And that's, you know, many theories of semantics are basically talking about certain patterns of physical causation. So there are things that happen in the world, and all of these chains of causation end up materializing well, not materializing, but end up being in your mind.
And those those links form semantics and how you understand the world.
And he's pointing out this Chinese room argument as an example of the deterioration or the you know, how those links can be broken.
Now, our friend Jez in the discord server, he said, well, Searle's argument is not a theory of semantics.
So what's the what's the purpose of this?
It's not telling us anything about semantics.
I want to learn about causal theories of semantics.
And Searle is just giving an example of why he thinks this particular situation doesn't have semantics. And then I think he pointed to Nagel's bat. You know, that that that wonderful essay from Thomas Nagel. And of course, that's not really a theory of consciousness either. That's a very similar thing.
It's just talking about in a particular inconceivable situation, um, how we wouldn't be able to understand what another conscious experience is, but it's not really talking about what consciousness is.
Um, the other thing that that John Searle is misrepresented on is people call him a dualist, and he's not a dualist.
He's he's a physicalist. He thinks there's one ontological substrate. And maybe we'll get on to that in a minute. But it's, um, yeah, I think he's misrepresented by a lot of people. Yeah.
I think, um, I think I'd rank him up there right next to, to Chomsky in terms of being misunderstood and misrepresented.
And, you know, you wonder sometimes about the sociological phenomenon that that caused that, that caused that to happen.
But I think, um, yeah. And jazz, um, um, I love jazz.
Having him in the, in our discord is a real, uh, you know, privilege for us because because he's just a precise thinker and seems to know, like some ungodly amount of, uh, philosophy knowledge.
And I, on the one hand. Okay, I, I finally was able to to see his point. I don't think it's fair. I see his point, which is his point is kind of like, okay, there's the Chinese room argument, but it has all these, you know, problems and it doesn't really provide a theory. And like, you know, it's it's almost like he's saying it's a teaser. Um, but and therefore it's not really worth that much because it's just a teaser for where, like, where the real work, you know, needs to be done.
But but I think you have to understand that a lot of times, I mean, that philosophical article. And by the way, Tim and I are big fans of the Stanford Encyclopedia of Philosophy. Right.
And even it says and many philosophers that are that, let's say, oppose, uh, maybe the, uh, the resulting intuitions of the Chinese room argument admit that it's one of the most important arguments ever made in philosophy. Right? I think there is a joke where where, uh, somebody and I'd have to look up the reference, but it was hilarious where he said, you know, basically the field of cognitive science can be defined as the field which tries to prove the Chinese room argument false. You know, so so I mean, yeah, it's an existential. It's had a huge impact. Yeah.
And so I don't think it's fair to just ignore that.
Like it's a very good argument because, um, or there's a sense in which it's a good argument because if you tell it to a person, it allows them to kind of immediately and intuitively grasp that there's a question we need to answer, right? And a lot of times I've heard it said that that philosophy is really one of the biggest tools we have for deciding what questions do we need to ask? Where are the unknowns?
Like where are the holes that we need to poke at?
You know, the threads that we need to pull? Right.
And so the Chinese room argument makes it viscerally clear that there's something we need to we need to think about here.
And actually we should we should say what the Chinese room argument is.
Do you mind? Because, yeah, I'm sure we have a lot of listeners who may not may not know it.
So just I'll just, you know, be brief about it, which is, you know, John Searle says, imagine, imagine there's this, you know, room, okay. And inside the room is a guy with a bunch of books and, you know, rule books and dictionaries and sort of all kinds of things, okay, that that allow them to map, um, uh, sentences given to them in Chinese. Okay.
Into answers or other sentences given in Chinese.
So, for example, a Chinese person can walk up to this room and it's got a little squawk box or a little, you know, thing.
They can slip a note under the door with, with some statement in Chinese or a question. And he'll do this procedure, this algorithmic procedure, to produce a result.
And he'll give it back. And this room, imagine that.
It's just, you know, it's it passes the Turing test or whatever.
Like people slip notes under there, they get them back and they think that these are coming from, you know, from a person who really understands Chinese. And his point is, it's obvious that this room doesn't understand Chinese. It's a guy who has no clue anything about Chinese. He's just performing this sort of mechanical algorithm. If somebody slips under the door or the crack, they're a joke. Okay, he's going to translate it into, like, you know, some answer like, ha ha ha, that's really funny. He's not going to be laughing at all.
He's going to have no understanding that that was a joke. Right.
And so this argument is immediately obvious to people, which is, yes, something's missing here. Something is missing in terms of understanding. And that setup, that Chinese room setup has been used to explore many other, you know, cognitive, um, cognitive aspects like consciousness and tension and intentionality, etc.. Right. Um, I mean, yeah, I think it's a great argument for, for bringing about that intuition that there is a question that needs to be answered. We'll get back to the decomposition of kind of cognitive functions in a bit, because that's very interesting, because in the original paper he was talking mostly about intentionality.
But when you read his later work, understanding the original paper was all about understanding. Yeah, yeah. Oh, I think it was intentionality and understanding, if I remember correctly, but I only recall it in a minute. Yeah, but but my point is, is that you can actually you can swap those words out for things like agency and consciousness. Certainly there's a wonderful talk which will reference now on the screen, which is John Searle talking at Google about maybe 12 or 15 years ago.
And I do recommend that you folks at home watch that talk if you actually want to understand what John Searle is talking about, because he's he's super duper clear in it. And one thing that struck me in that talk actually was it was very, very instructive.
He said he knew he knew that his dog was conscious not because of its behavior, but because of its mechanism.
It had eyes and it had a mouth. Right.
And it was a mechanism much like his mechanism.
And of course, the broader point of Searle is that it's physical mechanisms which give rise to patterns of physical causation, which give rise to our consciousness because to Keith's point a minute ago, the reason why Searle in the room didn't understand was because there's this phenomenal, which is to say there's this conscious experience, possibly of of laughter. And if you break that causal patterning, so if the conscious component is, is not there, then of course you wouldn't have the complete understanding, therefore you wouldn't understand. So it's very interesting to kind of distinguish this kind of function and behavior from the mechanism.
Because when we spoke with Murray Shanahan, he said, well, I can feel, you know, something is conscious if I share a physical environment with them and the behavior is kind of intelligible in a certain way, then I might intuit that that thing has consciousness.
And of course, this is quite slippery, because then you could start talking about a large language model as if it had consciousness based on based on its behavior. So Professor Jay Mark Bishop, he wrote a very interesting I think he was the editor of of a review piece on the Chinese room argument. So he's been studying the Chinese room argument for about 30 years. And we had him on the show a few times. And I must admit, I think the first time I had him on the show, I didn't fully grasp this thing because I think this is a thing that takes years of kind of reflection to actually understand. And certainly I think by the second or third time I had him on the show, I was starting to grok what he was talking about. But his opening line, I think on the second or third time he came on was this.
He said, indeed, it's my belief that computation cannot realize understanding deep mathematical insight or consciousness, and hence that there exists a gap, a humanity gap between that which can be achieved by computational processes alone and that which can be achieved by humans, potentially using computational tools.
Three A priori philosophical arguments.
John Searle's Chinese room Lucas Penrose's argument, and his own minor contribution to the debate. The dancing with Pixie's reductio ad.
We'll talk about that in a second. Underpin this claim.
He said that if any of these arguments hold, then an unbridgeable ontological gap between computation and mind is firmly established.
Well, aside from a diet of too much dodgy sci fi, he said he believed a deep confusion between epistemic concerns for example, how we as humans might establish if a sister or brother had understood our words or was feeling pleasure or pain, etc. an ontological, for example, how we might actually instantiate consciousness, pleasure, or pain in a machine lie at the root of this widespread, quasi religious belief that computation can actually realize cognition. He said.
He suggested that any commitment to what the American philosopher John Searle famously termed strong AI the view that suitably programmed computers can actually understand natural language, or generally instantiate or genuinely instantiate any other mental capabilities of the humans whose behavior they mimic is quasi religious.
And he went on to talk about, you know, uh, computational processes can never make us milk or produce energy from sunlight or sadly, make us rich. So, yeah, beautiful stuff. And I agree with you, by the way, that I think the first time we spoke to, um, Professor Jay Mark Bishop, maybe I wasn't prepared either to really fully ingest what he had to say. So it's nice to go back and read it with a new a new lens and and understand things, understand things better. I think, um, and this is a, this is a problem with, with the language too, because, um, Searle and I know this is going to sound weird.
I know this is going to sound really weird to us as like, computer people. Okay. But he distinguishes between compute and machine. All right, so when he's talking about a computer, what he's talking about is, um, a kind of machine that doesn't have that sufficient causal structure, okay, to produce understanding or consciousness or whatever. Okay.
He is absolutely not saying that there isn't a machine that we could build from some non-biological stuff that could have that causal structure. In fact, like James Bishop's statement is maybe even slightly stronger because he's he is really focusing there on the on the human right and saying, and you could almost read into that, that we'll never be able to produce that in some non non-human, you know, machine.
I think we're definitely far away from being able to do that.
But but Searle is saying no, we can like we can we can build a machine that can do that. It just has to have this necessary causal structure. It has to have this realization with this necessary causal structure. And um, both of the, you know, both Searle and, and I think even almost kind of let's say more beautifully there. Um, Jay, Mark Bishop are getting at something which is really fascinating to me, which is, you know, on the one hand, we have these beautiful equations and abstractions which which do really well at, uh, mapping to and explaining and predicting stuff that happens in the physical world. Right.
This is the, the, uh, remarkable effectiveness or the unreasonable effectiveness of mathematics, if you will.
And yet, on the other hand, we know like we know in principle and in fact, that these abstractions and equations do not capture the whole story. You know, a mathematical model of a black hole is not a black hole. A weather simulation is not weather.
Okay. A computational process simulating the human mind is not the human mind. There's this gap.
There's this gap between our abstractions, between our computations, between our math and physical realization. And it's fascinating. And I think this is where it's really largely unknown. And there's so much work to do there to better understand that. But I think you, you know, you have to accept that there's this gap. I mean, the arguments for that are just too strong. Yes. So the two major components to and by the way, I agree with you that Bishop probably is actually taking a stronger position than than so did. But we'll come back to Bishop in a second. But yeah, so Searle is saying that we need to have physical realization in order to maintain with, with a necessary causal structure, with the necessary causal structure.
Yeah. Precisely that. So the core of it is there needs to be a necessary causal structure. And he's saying I believe that some kind of physical realization is necessary for that.
There's another component to his discussion, which has led to accusations of him being a dualist, and that is basically this, that he thinks that the correct physical instantiation, and therefore the correct physical causal structure gives rise to consciousness.
And he's a materialist, so he only thinks there's one physical substrate, but he he makes the case that, um, this, this consciousness is subjective. Now, of course, we can refer back to Murray Shanahan's view on this. This is capital s subjective, but not in a dualistic way. So it simply means that you as an observer, you do not have privileged access to what is in my consciousness. And that's part of the reason why he thinks that when we kind of simulate, um, cognitive processes and machines, because they wouldn't have access to this special subjective experience, they would not have the full fidelity of understanding.
Now, I don't think so explicitly said this, but I think he's basically making the argument that our consciousness strongly emerges from a certain physical instantiation and causal structure.
Maybe you can explain what that means. Yeah.
And this is this is a super controversial, super controversial topic here. You know, strong, strong emergence.
Um, you know, is there such a thing? Does it exist?
So I'll put my cards on the table and say that I believe that both weak emergence and strong emergence, um, are are real phenomenon.
So let's let's talk about what those mean.
So in general, what emergent means is that, um, you know, if you have a bunch of parts, okay, like a bunch of molecules, for example. And let's suppose these molecules follow, you know, laws like the laws of statistical mechanics, for example. And you and you put them together that it's possible for the whole the sum of enough parts to exhibit behavior dynamics, if you will, that are just fundamentally different, um, from what the individual parts do. And, um, strong emergence would say it's possible for them to exhibit behavior that is not even derivable from the behavior of the parts. Okay. Now, like modern physicists, I don't know what I haven't done a survey, so I don't know.
But, you know, if you talk to most of them, they're they're really strongly are the ones who are vocal, let's say online are very strongly reductionist in the sense that they're like, nope, you know, the standard model all like, all I need is a standard model and I'll be able to like, compute everything and derive everything.
It's that they're really they really can't.
They can't do it in any remotely practical sense whatsoever. Okay.
Like the poor quantum mechanics guys like, they can't even really compute past what helium, you know, when they when they try to go beyond like too many, too many particles, they just get totally slammed by computational problems and they'll come in and say, yeah, yeah, okay, fine. Like none of my computers can do it.
But in principle, I could, you know, in principle I could.
And the fact is, even in principle, they're like all the theories we have, all the effective theories I've ever seen, they at least have points where they break down, okay. They have singularities that crop up when you try to apply, like, you know, the integrations or the effective theories. And as soon as you have a singularity, what comes out the other side is, is undefined.
So in my view, I think it's true that there are both computational barriers as well as in principle, analytical barriers that we cannot cross, we can't get from this side of parts to that side of the emergent behavior. Right? I think that this is possible at least. So if you take something like, you know, a good example is, you know, fluid mechanics, like the the behaviors of vortices and waves and the, and the, the giant, you know, red spot on Jupiter and things like that.
And then you keep going up and up to life itself, you know, life as an emergent phenomenon or gliders. And in the game of life, these type of, you know, little things that that stay stable and move around.
And it's like what Friston talks about, things that have these Markov boundaries and these these sort of stable dynamic attractors that that form like a thing, you know, that can kind of persist, persist over time. Um, all those things are definitely weakly emergent. Some of them may even be strongly emergent, and a lot of people who are even opposed to to strong emergence in general when faced with consciousness, like, you know, maybe accept that, okay, fine. That's the only strongly emergent behavior. Like, I think we Chalmers said this to us, right. He said like, yeah, I'm not really sure about all these other emergencies, but but consciousness may be maybe the only example of a strongly emergent behavior. Yes.
Quite an interesting analogy with Searle.
I mean, of course they have a completely different ontology.
I mean, Chalmers has a has an information ontology.
I mean, the weird thing about Searle is he's actually quite no nonsense.
I mean, if you actually take the time to understand what the hell he's talking about, it's it's surprisingly no nonsense.
And you said something very interesting there, which I think we should say for our agency discussion. But I believe that agency is also physically realized. And there's a distinct difference between the type of agency we have in the real world and simulated, but maybe we can save that. But the other thing you said, though, is there are many boundaries of subjectivity, and this is very important for building AI systems. Wolfram is hinting at it when he talks about how we are computationally limited as as observers. This argument from Searle is another great example of that. And this is all just pointing to how it might not be possible in principle to simulate many, many things in a computer. I wanted to touch on Bishop's Dancing with Pixies reductio before we move on.
So that was that was his main contribution. Can I just go on?
Can I jump in with two? Can I just jump in with two things?
So, uh, in preparation or actually not in preparation for this, but as a result of that debate in our discord, I watched that that Google talk, Searle's Google, you know, Google Talk, which I hadn't watched before. And I agree with you. There's this, this element of no nonsense pragmatism to him because I didn't know the story he told at the beginning where how he got down this whole rabbit hole in the first place is, um, you know, I guess he was maybe leading a research program at some point long ago, and he kind of walked over to the, uh, like, let's say the, um, you know, the cognitive science team or whatever. And he's like, hey, guys, what's up with this consciousness thing? How come you guys haven't solved this problem already? Right? And they're like, yeah, no, no, no, no, we're not really working on that because it's not science.
And that kind of set him off on this path. Like, wait a second. Why?
You know, why isn't this science? Like, what are you talking about?
Uh, and, and I wanted to also comment on, um, Stephen Wolfram. I really.
And what's the phrase he uses? It's, uh, computationally bound.
Right? Isn't that the. Yeah. The terminology, his observer sort of computationally limited computationally.
I think it's such a cool, um, framing of it.
It's it's really interesting, you know, saying how anytime obviously.
Let's suppose for the sake of argument that.
well, certainly the universe is manifesting a computation.
You know, it's unfolding as a computation.
I even think it's a hyper computation.
But let's suppose it's not a hyper computation.
Let's just say it's some Planck scale or sub Planck scale, you know, pixel simulation. Like whatever. The point is, it's this.
I mean, you can't even fathom how complex that computation is.
It's obviously so many orders of magnitude.
I'm not even sure higher than the computation that's happening in your brain. And and it's fascinating to me at this point that Wolfram makes that so many of the things that we, the abstractions we create, the the laws that we find, the science that we do, the behavior that we observe is due to this fact that we have this much, much less computation than the phenomenon that's unfolding. You know, before us, there's it's like they're compression artifacts. Like in a sense, in a sense, all our scientific theories are compression artifacts.
I mean, to me, it's just fascinating. And, uh, you know, that's why I'm always excited to see what he puts out, though, every time, every time Stephen Wolfram publishes something like some article or blog post. By the time I get to the end of it, because it's really massive, I get to the end and I and I'm reminded of like how little I've accomplished compared to him.
Like, how does he produce all this stuff?
I don't know. Yeah, that reminds me. There's a really good article by Stephen Wolfram on recursion and the last third of the article.
He's citing Gödel, Escher, Bach and Hofstadter.
And I think there was actually a formulation of this in Gödel, Escher, Bach, like a simple recursive function and studying how it could give rise to these structures. It's actually kind of similar in principle to how a cellular automaton works.
And then Wolfram did a whole load of different examples of this, And he kind of saw there's this there's this branching factor.
And sometimes you get cycles back and sometimes you get these amazingly beautiful patterns. And then they weirdly stop for no apparent reason. And there's this whole space of computation, which is fascinating. I don't want to spoil the surprise on that, because Wolfram is actually going to come on, and we're going to discuss that with him. But it also has echoes of our fascination with recursivity and the recursive mind.
I think there's a lot to be said for that.
And also, Keith, what do you think about this concept of, um, you know, computational subjectivity, given that we are Turing machines?
If you think we are Turing machines, then why, in principle would there be computational subjectivity? Well, at the moment I have a very simplistic view on that. And and again, for anybody listening, you should always take um, my simplistic views is, is potentially just ignorance, right? Like, maybe I haven't thought deeply enough about something, and so I've reduced it.
You know, I've done the reduction thing that I sometimes rail against.
I've reduced it to something simple for now.
It's almost like a placeholder for me.
It's like, okay, I got this simple thing for now.
Maybe I'll dive into it, dive into it later.
Okay, so so for for me, at the moment, I kind of agree with, um, at least what I think is a fair representation of something Karl Friston has said. And, you know, a blog post or, or somewhere, which is that let's suppose you're a let's suppose you're a machine. Okay. Let's suppose you're a machine and you're running a computation. All right.
And you start off without any sensors or any, any data whatsoever on the computation. You're performing itself like you don't know how much memory you're consuming or, or what the temperature of your CPU is, or whether or not a particular loop has been running for a long time. Like, you just don't have any kind of telemetry on your own computation. If you start hooking that up. Okay.
Like you start having data and knowledge about yourself and the computation that you are performing right now, right?
Then when you get to a certain what Friston calls temporal and counterfactual depth, okay. Once you have enough.
Once you have data from your own internal activity processing causation that spans enough time, okay.
Both in the past and the future, like your ability to kind of forecast out what you're going to do. And once you have enough counterfactual depth, once you're able to think about alternate paths and alternate paths that you could have done in the past, that that's, you know, once you get far enough along that depth, you become self-aware. You you gain, you know, consciousness, um, uh, and maybe some other, you know, cognitive aspects.
Yeah, we should probably save some of this for our agency discussion, but we did have a bit of a slightly different interpretation on that.
I remember because I felt that he was talking about the depth of nesting of the Markov blankets, which give rise to a kind of counterfactual depth of planning. So his thing is that you you are more of an agent when it's as if you are planning more steps into the future.
So your planning horizon increases. And when that happens, can we ask him, could we? Because maybe we are so fortunate to have. Yeah, I mean, we're so fortunate to have him as a friend in a contact. We we should maybe email him and see if we can record like five, ten minutes of him explaining this to us. That would be amazing. Yes. But even that, I guess, is a form, if you think of it computationally is some form of of recursivity.
But he's saying that there's some kind of goal.
It's a form of depth. Depth. Yeah, yeah. Well I mean the depth.
So the depth can be nested. So I mean, I see what you're saying, that they're so I don't know if the nesting if he thinks the nesting is a necessary component, but certainly it could be it could be nested, you know, if you have enough, if you have enough depth.
And I guess in some sense to it, it almost has to be because, you know, since since your mind, you know, one of the most efficient ways in which you can do depth of computation is recursion.
And so it probably is nested, but it would be really interesting to get his take on that. And the degree to which he thinks nesting is a necessary component versus just an optional, you know, an optional or sufficient component. We need to get Freston back on.
I think what we need to do, Keith, is I want you here.
We need to get you in a room with friends. Okay.
Oh, I would love to meet him. We've got a we've got a the thing is, I mean, maybe I can just go in and film at his place and we can do it over the internet or something. But I would love for us all just to be in a room sipping a sherry, as you say.
And discussing the meaning of the universe. Yes. So.
But anyway, we'll save that for our agency discussion.
But there was one objection from Cog champ on our discord, I seem to recall, and this was quite an interesting objection.
He was saying, yeah, yeah, we need the physical causal structure.
And he was saying it's like an oxymoron to say that computers don't have causal structure. So, you know, we have all of these algorithms. I mean, there are books written about this, you know, like that. The algorithms are affecting us in many ways. So they are somewhat, you know, maybe not agential, but but they do have causal power on us.
So if there is a space of causal power on us, why could we not shape that space with the right algorithm? Well, I think, I mean, I think maybe their Cog champ and, and Searle and possibly us were talking past each other because Searle is saying it has to have a certain causal structure, not just that any old causal structure will will do the job right. And so sure, computers, you know, a computer is a kind of machine. It does have a causal structure when it's performing a computation. There's obviously a causal structure there. Otherwise the computation wouldn't work. Right. Like, you know, the the transistor charges cause this the clock cycle causes that.
I mean, so there's a causal structure.
I think, um, what Searle is saying is that you have to have the right kind of causal structure. There's a particular causal structure. And he doesn't specify this, by the way. And this is kind of the weak point or as far as I know he doesn't specify this.
This was the weak point that Jeremy and Cog champ, you know Jez and Cog champ point out. Right. Which is like okay, there's a causal structure there. Like why is it, you know, or he hasn't told us what the causal structure is totally fair.
I mean, it's Searle's claim that there's a certain kind of causal structure. And so I think it would be highly unlikely, highly improbable that the causal structure that we build into a computer, into a digital computer for purposes of, you know, carrying out a calculation, I find it like pretty, pretty astronomical that that would just happen to also coincide to the causal structure necessary for understanding intentionality, consciousness, etc.. And this is part of what is brought out in the Chinese room argument. Like people intuitively know that a bunch of gears that you're sitting there like just churning to, to punch out, like, you know, the next conclusion to the sentence.
Like intuitively, we think it's missing something.
That's the whole point, right? It's missing the right causal structure. Okay. So I think a fair interpretation then is Searle isn't completely ruling it out.
He thinks that there is something quite special about physical real mechanisms. Right. And that is likely to give this rich causal graph that couldn't, you know, that's not likely to be replicated with any algorithm on, on the internet or anything like that, because it does bring us to, to the next adjacent point, which is um, there are um, you know, linguists like Steven Piantadosi and Steven's written many papers saying that Chomsky was wrong and that language models do learn semantics. And he's got this theory, which is called concept role semantics. And Jez is very interested in this.
And, you know, essentially it's saying that there is some kind of causal structure, but it doesn't actually need to be as physically materialized as we think. And there are Internalist and externalist versions of this, because there's always the problem with having an externalist causal semantics, which is to say, well, I can imagine abstract concepts, I can imagine unicorns, I can imagine things that don't necessarily have any analogue in the real world.
And of course, you could go the next step and you can say, well, maybe they are composed of primitives which do exist in the causal world, but there is a whole school of thought that, you know, in a computer, if we just represent this causal semantics graph to high enough fidelity, then the understanding threshold will be met.
Um, honestly, I at this moment, I'm not prepared to rule that out.
Like, I don't know that I can say that's not possible for understanding. Okay. For for understanding.
Um, yeah, I can't I mean, I can't rule that out at this moment.
I think, um, you know, there has to be a certain causal structure, whether or not we can get to that causal structure by virtue of just a special kind of program, like a complex enough kind of program. Um, could be possible. Um, and then if and if it even goes as far as to, uh, consciousness, um, well, then we're in a, you know, interesting situation, right? Because it's like, okay, I've got this computation now, um, which has somehow met all the causal criteria for, for, uh, consciousness. And I suspend the calculation.
And so it's just sitting there stored, you know, on a, on a, on a disk drive. Now there's a bunch of, like, bits, maybe even gzip it. So I've gzipped this file like that file definitely no longer retains the causal structure. Right.
Because it's been compressed and it's not running right.
But I can rehydrate it as, as now a conscious, you know, conscious being.
So that'll definitely open up a bunch of interesting territory.
I mean, I think clearly definitely. Um, Bishop, I think, is arguing against this possibility if I understand him, you know, correctly.
Um, I think Searle to, um, you know, probably thinks this is not, not the case, right? Um, I don't know. I think it's okay.
That's a tough one. Another thing is, I don't think Searle explicitly has a position on internalism and externalism, but I think people assume that he's an internalist because he talks about the brain. I think he believes that the brain alone, you know. So you have the brain in a vat.
And because of the special physical mechanism in the brain, that brain would be conscious, which means there'd be some subjective component. But based on what he said in the Google talk, I think he is um, I guess this would be technically embodiment, but there are many different brands of or, you know, flavors of externalism which are pointing to there being cognising elements outside of the brain and maybe the body and so on.
He said that, you know, his dog has eyes and a mouth, and that contributed to this mechanism of it being a conscious being.
So I think it's reasonable to assume that he's at least open to the idea of there being like an external component to cognition.
Mhm. Yeah. I mean maybe. Yeah. That would be an interesting question as well. I think um, I think my reading of it, and this is just my reading of it, is that he does require a certain internal causal structure. Like without that internal causal structure, you're not going to have consciousness or understanding.
And then like you said, I think, uh, maybe he doesn't require externalities, external, you know, causation, but that but that there can be a part of it and, or they can be a part in, uh, of, of our ability to know or to infer that something has, um, you know, consciousness. But for example, I would bet just from having read a lot of them and looked at, you know, videos and whatnot, that if you were to put the question to him, does a person, um, let's suppose a person isn't necessarily in a coma, but they're somehow totally shut off from the outside world, but their brain activity is still is still normal.
You know, it's like a dream type of state, right?
Um, I think he would still argue that that person understands whatever language they learned natively, that they're that they're even conscious that, you know, um, despite the fact that they currently have no external connections really whatsoever. So J.
Mike Bishop, his contribution was this dancing with Pixies, Pixies, reductio and uh, and this. I'm just reading a summary of it here, but it's a critique of computational theories of consciousness that builds on Hilary Putnam's ideas.
So the key points are one. It targets claims that a suitably programmed computer can instantiate general phenomenal consciousness, or qualia, purely in virtue of executing a program.
Two it argues that for any computational system claimed to have phenomenal states due to its program execution, we can map those computational states onto state transitions in any open physical system, like a simple counting machine or even a rock.
And three this mapping can be done for a specific execution trace of the program with defined inputs over a finite time interval.
Four therefore, if the original computational system has phenomenal experiences due to its computational states, then so must the mapped physical system, which leads to an absurd, he says, form of panpsychism, where consciousness would be ubiquitous in the physical system, little pixies dancing everywhere.
The argument aims to show that computational or functional organization alone is not sufficient for phenomenal consciousness.
Now, is it such a reductio ad absurdum to say that if we map a computational process to an open physical system, then the open physical system would be consciousness, and that's an absurd thing. Do you find that absurd? I yeah.
So I mean, like Putnam's Rock is sort of the, you know, the, the original version of this. And yeah, I think it is absurd to say that a rock is conscious, but I think so.
There's a couple little loopholes or, you know, not not the rock, the, the, the overall, you know, organization. Would that be conscious?
Well, but this is specifically about, I mean, the Rock or in the case of dancing with Pixies and open and open system. Right.
Well, so he's saying take a cognitive process in your brain and represent that as a series of like hose pipes and water and paper cups and whatnot.
You know, just just sort of just map it to, to some open physical system.
So it's performing the same computation.
And then let's imagine that that thing in the physical world that we've just mapped it to is also conscious.
Does that seem absurd to you? Well, I just want to clarify here.
You can do that same mapping to a rock.
That was the whole point of, you know, Putnam's Rock is that you could even map it to forget about, you know, nice pipes and and hoses and gears and things that, you know, maybe are cool.
Like, you can even just map it to like, the the sort of atomic dynamics of a sufficiently large rock right to some subset of its dynamics.
Um, and yeah, I think that's absurd. Like, it's absurd to think that that the Rock has consciousness. And I think it's like somewhat, um, absurd. Absurd to think too, as well, that, you know, a bunch of gears and like, whatever.
Of, of an arbitrary structure that we mapped it to necessarily have consciousness. Right? Um, we are for Estonians, right?
We're for Estonians. So. So Friston says that there exists.
Certain dynamics and functional organization which give rise to consciousness. So if we. If we say yes to that, then why do we say no to mapping a cognitive process to an open physical system?
Well, I mean, so I'll tell you why. Like I'll tell you where the I'll tell you where kind of. The, the problem boundaries are in, in that argument, which is that one one thing I really like.
About the dancing with dances with Pixies reductio ad absurdum is that it does lay out very precisely the argument and in fact lays it out so precisely that its own weak points are explicitly put in there. Okay.
Which, as it says, if we if we pay careful attention to it, that for any finite period of time you can do this mapping. Okay.
So like you can choose some finite amount of time a day a year, like a whatever. Right. In principle you can do this mapping.
Here's the problem is that in whatever thing you choose and you do this mapping, suppose it's an hour okay.
It an hour plus epsilon okay. An hour plus one second.
Those two systems will have diverged unless they have exactly the same causal structure. And this is kind of the key of what you know, Searle is pointing out is like sure I can take you know, let's suppose let's let's boil it down to like the absolute vacuum state of this problem, which is a Boltzmann brain in a vacuum, and a Putnam a large Putnam rock in a vacuum. Okay.
And so if we all say, yeah, the Boltzmann brain has consciousness, you know, it's conscious, it understands, it's, you know, whatever else. Okay. And then there's the Putnam Rock and the vacuum over here, and that the Putnam Rock is large enough.
I can map all the dynamics of the Boltzmann brain for any finite period of time to some subset of dynamics in the rock. Aha!
The rock is, you know, conscious. No, because outside of that domain epsilon time, beyond that, they diverge. Okay.
So the computations are not are not the same.
The dynamics are not the same. The causal structure is not the same.
And that's the whole core of of Searle's argument. Right.
And it's and it's the whole it's the sword that cuts through this idea that like just the, just the observable functional. Functional.
Functional behavior is enough. Like you have to have the causal structure built into the realization. Um, but yeah, I think it is absurd to think the rock is conscious. Yeah. I mean, I was I was just going to say that. Yeah, I intuitively agree that, you know, it's um, it is a reductio ad absurdum because you could just take a trace of a, of a cognitive system and then it would diverge at epsilon one or like 0.0001. And then there are all of these intellectual exercises of, of the fidelity of, you know, maintaining the causal structure. So, you know, what, if we have a Chalmers did this with the combinatorial state automata, you know, so we model the counterfactuals and we model some underlying process. And, and eventually maybe there'd be some threshold level where we would capture the phenomenon.
And, and that that's very interesting philosophical territory.
But certainly if you just capture a trace. Then.
Then of course, it wouldn't. It wouldn't be the same thing. Yeah.
I mean, I agree, I think it's fascinating. It's interesting.
I think these are very deep questions.
I think they're and that was the power of the Chinese room argument is that at least laid out this this in question in a very visceral way.
You know, you you hear the problem and you're like, yeah, there's something we need to think about here.
And so it's it's definitely helped to spawn a great amount of research.
Um, yeah. All fascinating questions. You know, I wonder how many we'll see answered in our lifetime. Uh, yeah, I don't know.
Final thought. What do you what do you think about Hinton winning the Nobel Prize? Um, look, I you're drawing for breath after I asked that question. Yeah, because I don't I'm not.
I don't think I'm qualified to to have an opinion.
Honestly, um, I don't think I'm. I don't really think I'm qualified to have an opinion because I haven't, I don't know.
You know, I haven't surveyed enough the whole literature and everybody's, um, contributions to it and and whatever.
So I just don't think I'm qualified, actually, to have an opinion.
I did see I did see a lot of, let's say, some of the controversy, um, around that. And, and the one thing I do know is true is that academia is plagued by politics, okay?
It's plagued by politics and corruption and pettiness and gatekeeping and, and, uh, you know, power struggles and turf fights and bad behavior and people not siding when they should and stealing work.
I mean, so it's absolutely the case that academia has a lot of problems. Okay. Whether or not any of those are relevant at all for for this. This choice.
No, I couldn't say, I can't say. And so therefore, like, I just I take, I take the award at face value, which is congratulations to them.
I mean, they've obviously had huge, uh, contributions um, to, to the field of, of machine learning. And, you know, um, I would love to see any of the, um, the folks that we've talked to on the show or interacted with these kind of like, let's say, the godfathers of, of AI. I think they're all massively deserving of, of accolades. So it's all I can honestly say.
I agree with that. His his contributions are undeniable.
But, um, as for the credit assignment thing, no need to comment on that because we're going to release part two of Jürgen Schmidhuber.
And he was pretty scathing, actually. So you can only imagine what he said.
So I won't comment. I'll just let you folks wait until we release that. But the the other controversy was it was a Nobel Prize for physics. And this is kind of related to the whole Chomsky comments about, well, language models aren't a theory of, you know, they're not a scientific theory.
They don't tell you about what the thing is and what it isn't. Yeah.
And to what extent is deep learning like, what has it got to do with physics, if anything? So again, I don't I don't feel that I'm qualified to decide what should and shouldn't count as a physics Nobel Prize, I certainly know. So we interviewed Daniel Roberts and Sho Yaida. Right. And they had this beautiful book that I really liked a lot, the Principles of deep learning theory, which in that in that case, it's the reverse.
They're applying physics, um, you know, ideas to the study of deep learning. And it struck me as totally legitimate, useful, uh, something we should do more of, So I can't say it's impossible that that, uh, machine learning work can't, can't actually be physics or or considered physics or contributing to physics.
I just I don't think I'm qualified for this.
These are these are way above my pay grade, um, to to decide what is and isn't, you know, counted as physics. Honestly. Yeah.
But do you accept the analogy to Chomsky's comments on language models, which is that deep learning models are not a scientific theory in the sense that they don't provide a powerful explanation.
Well, but Chomsky was talking about, you know, a specific I mean, what Chomsky was saying is that if you use all these techniques, deep learning or whatever, and then you train a language model, it doesn't tell you anything about language.
It may tell you something about physics, though, like because because the physics was in the in the training process was in the math was in the engineering and the science that went into the process that trained it. So it's not the it's not like somebody. It's not like Hinton went and trained a, a large physics model, right. Like a large quantum mechanics model.
And then he got the award for physics.
It was it was more that the award came for the work that went into the the techniques and the math and the science behind creating these models and training them. Um, yeah. But the naysayers have said that the techniques that go into deep learning from a physics point of view are quite, I don't want to say trivial, but, you know, they're they're not they're not that complicated. And indeed, people physicists have applied their knowledge to deep learning.
But does deep learning give us knowledge about how the universe works? Because I think that's what the whole pursuit of physics is about. Well, on that Particular score like that question. Like I'd have to say not in my opinion right now. I mean, I don't I'm not aware of any, you know, fantastic insights that, um, uh, I guess the only one I can think of is, is I think there's something very interesting about the fact that hyper overparameterized models are able to generalize at all and that they don't, that they don't just overfit into absurdity. Right.
I think that's really interesting. Um, and maybe that says something deep about, I don't know, the structure of, um, maybe that helps us explain the unreasonable effectiveness of mathematics, for example. Um, but again, I don't feel qualified to I don't I don't feel qualified to, to comment on this specific, um, award. Um, I think, I think there are too few new belt. There are too few Nobel Prizes to go around. Every year there are many more people deserving of a Nobel Prize than there are Nobel Prizes to give out, and so very difficult decisions have to be made, and I just don't feel qualified to second guess them. Another couple of angles and yeah, I mean, like we should just both say that we are, you know, we're allowed to have opinions even if we're not an authority on every opinion.
But another comment was made that Nobel prizes are given out almost speculatively based on the potential of of a discovery, because, of course, Demis Hassabis and a couple of guys got the Nobel Prize for the I think it was the drug discovery thing.
And this is certainly something which has huge potential.
But the naysayers are saying that this is kind of validating a lot of the bad things. I mean, apparently Google in a lawsuit, even this week have started citing the Nobel Prize.
They say one of our VP's and top scientists.
He just got the Nobel Prize. So what are you talking about?
That there's copyright problems. And I think Hinton is also starting to kind of double down and say, well, look, I told you so in in ten years time, these things are going to be doing everything and anything.
You think that humans are needed for now, they're going to be doing it.
And it's lending credence to this idea that there's just no end in sight for this technology. And let me say as well, that we all agree that this technology is revolutionary.
It's changing the world. It's changing everything.
People just disagree on how that is. Some people think it might recursively self-improve and its capabilities will just keep going up.
And it's really intelligent. And and other people just worry about the social impacts of it and it reducing our agency and exacerbating inequality and so on and so forth. But no one's going to deny that this is a real revolutionary technology. Yeah.
And the stuff you mentioned, I'll come back to that.
But what you mentioned earlier, like I said, there's politics aplenty, politics aplenty and, and academia and, and I cannot say in this specific instance that politics and corruption and other bad behavior played a role. Okay. But it's certainly exists in academia. As I've said before, that's part of the reason why I'm not in academia, because I really had a very difficult, you know, unfortunately, like, I'm kind of genetically a Boy Scout or something. And I, uh, I just had difficulty dealing with corruption that was wasting public money. Okay.
Like public money grants and things like that.
So I'm out of there, right? I mean, so, yeah, I can totally believe that it's being abused certainly by corporations to, to to get more money and power. Absolutely. Um, as for like, the disagreements about AI, yeah. You and I, we obviously we know that these are powerful Technologies. A lot of what we talk about, though, is that one, there are limitations to these technologies, and we're suggesting ways in which they can be expanded and further improved and powerful tools can be misused. And. And I was happy, you know, and earlier I didn't think lehron I to I would like to because it was a great opportunity to go on doom debates. And I thought the the feedback was tremendously positive. We got a lot of positive feedback of, you know, funny enough, it seems like it helped to humanize Dugger.
So I'm not quite, as, you know, evil, uh, evil as I used to be.
But but one concept that we came up with in that podcast is that we should be talking about a lot more. We should be talking about a lot more. I harm, okay, which is the harm that AI does. It brings lots of great benefits.
It's also bringing harm and it's being misused to cause harm.
And we have to think about that, and we have to wrestle that, and we have to reduce the harm that we that we accrue from, from AI and and increase the good. I agree with that.
I mean, we didn't speak much more about Leon's podcast, but at the beginning, I was surprised that you said your doom was 25%.
And yeah, I've always listened to the Sam Harris podcast, and there are lots of folks out there who are really interested in existential risk. Not necessarily AI, but pandemics and nuclear war and stuff like that. It's it's very legitimate.
I mean, Noam Chomsky talks about it all the time.
He always says we're, you know, like one tick away from doomsday talking about the threat of nuclear Armageddon.
This is all absolutely true. And I guess I maybe I should be worried. I accept it to be a truth. It's just not something that I'm personally, you know, that preoccupied with.
So that's undeniable. And then there's the sort of the intermediate thing of harm from AI non-existential harm, but just social harm. I'm completely on board of that.
And then, you know, if you go all the way down that line, you get to this existential harm from AI. And for me, that would require, um, AI to be essential. And I for many of the reasons we've spoken about today, I don't believe that that could happen even in well, maybe, maybe I'll say in practice, just to be safe for now, but certainly not something that we should be worrying about anytime soon. Well, the the nice thing from my perspective, is if we could just get people to really take seriously AI harm, okay, just take it seriously and look into it and try to mitigate it and try to prevent it, try to reduce it.
Um, a lot of the steps, a lot of the policy steps that we would take to reduce AI harm, uh, would actually reduce AI risk as well.
And if we started to take AI risk seriously, it would reduce the chance of AI doom. So these are like steps in the right direction. And the reason why I want to frame it in terms of or reframe it in terms of AI harm.
Is that something that you could get? I don't know, like the vast majority of people like 90 plus percent, you know, to to resonate with like, yeah, I mean absolutely AI harm is it's, it's reducing quality of life by, by leading to unemployment or people losing their jobs or, you know, no longer being able to get paid for the work they want to do. It's causing massive harm to our children, you know, through like, the social dilemma, um, type stuff.
It's it's sucking up, um, attention into, like, completely pointless exercises, like infinite scroll of, you know, content with no mental nutritive value whatsoever.
I mean, if we could just get people to almost all of them would resonate with some aspect of I harm. And then then we could start to take it seriously for policy, uh, for public policy.
The thing that I don't like is the conflation of AI safety and AI ethics. I'm trying to find the paper that Penny shared the other day. Here we go.
Yeah, there was a paper that I read by Dagmar Monnet called deconstructing the AI myth, Fallacies and Harms of Algorithmic Algorithmic Fiction, and we'll put a link to that on the screen.
And one thing that struck me actually is, and I think it's fair to say that Dagmar is on the left, a lot of discussion is around the reduction of our agency as a function of of AI. And many different philosophers talk about this. When I interviewed Luciano Floridi, he was talking about re ontology ization as we migrate to an information ontology. So we're basically living our lives in the infosphere and our digital identity is in the infosphere and we need technology to survive and so on. And and perhaps that's more of a consciousness argument, but but there are all of these arguments that are talking about various different components of our cognition, be it our agency or consciousness or whatever it is.
They're being eroded as we acquiesce to technology.
And, you know, so that that's I see that as a bit of a leftist argument, but I'm completely on board with that. That really makes sense to me.
Well, I mean, I think there are so many dimensions of, of AI harm as well as AI. Good. Um, that the whole point is we need to be having these conversations. And now a lot of times the conversations become really like quite nutty.
Um, you know, so I don't I don't resonate with any philosophy whatsoever that doesn't correspond to physical reality. Okay.
So like I start with physical reality as kind of my, my bedrock.
And I think that no matter how much we don't like truths, we have to accept them as as truths and then deal with them, you know, head on.
Okay, so but I think the nuttiness and conversations would kind of get worn down if there was a more robust conversation.
So we need more people having the conversation.
We need more voices involved, not fewer.
We need real data, real science, real consideration. We can't leave.
We can't leave the steering of AI to just the fringes of people.
We need to find a way to talk about it with, with everybody.
I agree with that, and I feel that Luciano Floridi in principle is saying that. He's saying as we become increasingly divorced from the physical reality, even from a legal perspective, the territorial reality, these these systems abstract so many of the physical boundaries that define how our legal system works, how we live our lives, and so on. But I did want to be clear.
I wasn't trying to say. I wasn't trying to say he was nutty.
That's what I'm trying to say. Oh, no, no, no, I know, I know, I know what you were saying. Voices. Yeah, yeah I know I'm getting I'm getting onto to the nutty voices you were talking about. So. So this.
This is why I don't like the conflation of AI risk and ethics.
And you kind of pointed out why on your earlier on discussion when you started talking about orthogonality and instrumental convergence. Right.
So there are all of these philosophical thought experiments which are used as justification for AI risk.
And I think that they are extremely speculative. Right.
It's fascinating philosophy. I think it's legitimate philosophy, but I don't think they mix particularly well, certainly in if you want to be doing policy. I don't think that those arguments should be factoring in in any way the same level of valence as talking about the other stuff we're talking about, you know, like the effects of social media and bias and fairness and deontologism and all of this kind of stuff. I don't want to use the word science fiction, but I think it's completely on the speculative side of things.
Well, I mean, I completely agree with you there, which is that and I agree with every point that you said. So, yes, there is a whole host of speculative, you know, things. It's really fascinating.
Definitely philosophers and and maybe, you know, chumps like us should be talking about it and should be engaging with it 100%.
And you're absolutely right. It's not what should be factoring into the policy, you know, but what I'm trying to get across to to folks who are concerned about AI harm, AI risk.
AI doom is you don't need science fiction and diamond nanobots and like whatever else in order to convince vast majorities of people to do things that you would like to have done that would help with those concerns, right? You know, there's always this conflict. People think that that, uh, that public policy should just be run by science.
They believe in kind of like a technocracy. Okay.
And that's not how the world works. And it's and it's actually contrary to a lot of economic principles and, and, you know, political principles and whatever else you've got to find, you know, because you're dealing with this massive evolutionary algorithm, which is the 8 billion people plus every other life form on Earth in an ecosystem.
And what you have to do if you want to steer that ecosystem in the directions that you want, is you have to be able to find arguments that resonate with large fractions of the population. Okay.
And so you need to focus on the core and the simpler arguments that will resonate with more people. Let's do that.
And if we do that, we'll be able to get better policies that not only will reduce harm, help to mitigate risks, they'll make us more prosperous, healthier, happier, um, overall, have more resources to figure out how to solve the alignment problems.
ET cetera. ET cetera. So, you know, we've got to focus on human flourishing. I guess at the end of the day, which I know you know, Sam Harris would agree with, right?
Like, we if we focus on human flourishing and reducing harm, um, we'll get a lot of the policies we need to handle these other concerns.
Easier said than done. Um, on on that note, Doctor Duggar, we will round off this Philosophical Steakhouse episode.
Let us know at home, folks, if you like this kind of thing.
Me and Keith just rambling. Makes an interesting steakhouse, I love it. Yes. Anyway, cheers. Cheers.
Up Next
Machine Consciousness Debate: AI, Minds & Future of Intelligence
@dmitry_volkov
1.7K views•2025-08-18
Lincoln Douglas Debate Format Explained by Coach Joe Vaughan
@nsdaspeechanddebate
97.8K views•2012-11-08
Should We Destroy the Machines? Big Tech Surveillance Critique
@TheStoriesWeTell303
167.9K views•2025-11-30
Logical Paradoxes: Identity, Semantics, and the Self
@Vsauce3
6.2M views•2016-03-08
Related Study Plans & Knowledge Roadmaps
Structured learning paths in Philosophy