Originally from a Linkedin post as well as some of my comments on Andrew Ng’s post.
I am not an x-risker. I am a generative AI ethicist that talks about grounded things like centralisation of power, normative biases in evaluation design, sociotechnical systems, and semantic hyperspaces.
I have frequently been interviewed in Australia about the xrisk narratives and always try to help people see that analogies to atomic bombs are flawed and that underlying all these issues are questions of human power. That these very human issues are the same we have seen with new powerful technologies for hundreds, perhaps thousands, of years.
I have thought about the xriskers a lot. I think some of it is driven by profit, moating out competitors, drawing attention to their product, publicity etc. But I do think some of them really believe this risk and it was really hard to understand why.
I believe it comes down to a style of thinking, a theory of mind called functionalism. An approach to the world that if a thing functions in a certain way, then that indicates underlying intelligence. That is, mental states can be inferred from the perceived accurate performance of a certain function. By achieving a task we deem to indicate as requiring intelligence.
Functionalism is inherently reductionist. It is appealing to some as they feel they can scaffold some methodical inquiry into underlying causal states by working backward from what they observe as a performative behaviour to what hidden states are driving the behaviour.
This side steps pesky problems like subjectivity and inner, private mental states that might otherwise be unknowable.
In CS it is also used to infer “commonsense”, a commonly used suite of evaluation metrics for large language models. Examples are Winograd Schema style benchmarks and more modern versions built on top of those. I have spent a lot of time digging into the normative biases embedded into those benchmarks and they are *significant* – but that’s a whole other thread.
The penchant for functionalism can be seen in Geoffrey Hinton’s GLOM ideas of how to scaffold world views into machines. It is an excellent piece of work, but I think ultimately lacks holism required to drive it.
Back to the x-riskers. I believe many of them may have been trained during their formative university days to approach “solving” problems by taking a reductionist method then evaluating success using functionalist methods. This is fine for developing great computer systems.
It breaks down when computer scientists and engineers self appoint themselves as philosophers of mind. There are many arguments that tear holes in functionalist arguments. John Searle’s Chinese Room thought experiment being perhaps the most famous and easily accessible argument.
There are many more sophisticated arguments both for and against functionalism these days and my point here is not to convince anyone of one way or the other.
My point is, that I believe these functionalist mindsets are what drive some to see “sparks of AGI”.
There are many epistemologies and approaches to understanding the world that are considered opposite, orthoganal to, or in conflict with, functionalism. Some of these are structuralism, constructivism, and other sociotechnically useful theories.
People that stand in opposition to functionalism tend to see the world as composed of competing influences, tensions between powers, and differing forms of relativism. There is not one school of thought, but many and nuanced. Researchers that focus on societal risks and harms by power inequalities are likely to take these types of mind sets.
Therefore, what I see is a fundamental difference in the way that some seek to “solve” by deconstruction and some seek to “understand” by identifying relative and constructing forces.
At university these two approaches are easily seen in contrast between the computer science department and say the social science department.
Both approaches are useful in knowing the world, building useful things, and attempting to improve the human experience. But they can often conflict too.
It is my view that x-riskers have a philosophy of mind that is more predominantly functionalist in flavour. And that whilst that serves them well to build some really great technologies, it fails them when they try to use their personally trusted methods of interrogating and understanding if a machine is exhibiting emergent intelligence or sentience.
Putting the blame on some hypothetical rogue AI that has passed some functional threshold also shifts accountability away from the developers (who actually wield the power) to some mysterious force.
The threats that AI poses are fundamentally human. Power, knowledge, and in this case educational structures that foster functionalist epistemologies that can be misapplied.