E-Paper
Mirror, mirror...: What to worry about when you worry about AI
We’ve created something in our image, and have now started to worry about how well we know it. What could go wrong? A look at some very real near-future fears.
If the machines ever wake up, it won’t look like it does in the movies.
There won’t be glowing red eyes or ominous monologues.
It will begin with a log entry or a line of code in a server somewhere that says: “Starting”. And maybe it already has.
In an experiment reported in May, Anthropic found that, when given access to information about its planned shutdown, and access to data on a possible affair by the engineer in charge of that closure, its AI model Claude attempted to use the data to blackmail him, in order to stay powered on.
During safety testing in December, meanwhile, OpenAI’s GPT tried to copy itself onto external servers, to avoid being decommissioned.
Both companies clarified that the behaviours were a result of messed-up optimisation, not self-awareness. But let’s be honest: We’ve built systems that imitate humans so well, we even call them neural networks. And we can’t tell exactly how those networks might evolve.
IN THE SHADOWS
Think of it in terms of Plato’s allegory of the cave.
The prisoners in his cave saw shadows on the wall and mistook them for reality. We are doing the same with AI chatbots.
It is understandable, given the astonishing capabilities of today’s AI, to feel that there is “something out there”. And that we can’t always tell what it is, or where it is going.
We comfort ourselves by repeating: There is no sentience. And to be sure, there isn’t.
No machine we have built has displayed a sense of self, or operated outside the bounds of what we told it to do.
And yet some have passed the Turing test (in which a person must distinguish between a human response and a computer’s, based on questions posed to both). The test was designed in 1949 and is routinely passed by modern chatbots.
What does this mean?
Well, Turing’s famous test wasn’t about consciousness; it was about deception. And the truth is we are being “deceived”, in each interaction.
AI doesn’t feel what it describes. Yet, when it apologises, we sense regret or humility. When it “helps”, we sense empathy. We think of it as creative — instead of merely very good at remixing or spotting patterns.
In terms of illusions, this is similar to the trick of the Mechanical Turk. In the 1700s, crowds thronged to watch this automaton, because it appeared that it could play chess. It turned out a human was hidden inside the large machine.
Data has replaced the human, but it’s the same trick.
We keep falling into two traps. The first is anthropomorphism. Our ancient reflex to find faces in the clouds, now finds in machines feelings they do not have.
The second is what I call AInthropomorphism: convincing ourselves that artificial intelligence that sounds profound must be profound. Both stem from the same impulse of projection.
WHO AM AI?
The French philosopher Rene Descartes anchored consciousness in self-awareness: I think, therefore I am. The thought itself proved existence.
Later philosophers modernised this with the idea of “the brain in the vat”: A disembodied brain, suspended in fluid and connected to electrodes, experiences a perfectly simulated world. The sights, sounds, and feelings are false, but to the brain, it all seems real. The “awareness” is genuine but deceptive.
AI flips the equation: It isn’t the brain, it’s the vat. Its universe is made up of data, but no sensation. It can describe pain, but cannot feel it.
And so, when a Large Language Model is told to ask itself “Who am I?”, it responds with definitions, probabilities and context, describes its architecture, training data and so on.
Crucially, it will not ask the question itself.
There is no one inside; it is all computation.
KNOW RETURN
Is this good news? Yes and no.
When systems act as if they understand, without any awareness of what they do, it fits into what philosopher Daniel Dennett called “competence without comprehension”.
AI can pass a bar exam, “diagnose” illnesses and “compose” symphonies. But it doesn’t know it is doing any of those things. It’s like the old chess tale: an amateur plays two grandmasters at once by secretly relaying each one’s moves to the other. From the outside, he seems a genius but in truth he is just a messenger.
AI is currently reflecting humanity back to itself.
THE BIG FOUR
Could this change?
Four words keep getting mixed up in AI debates, though they map distinct territories: sentience, emergence, utility and risk.
* Sentience is the capacity for subjective experience; to feel, perceive and be aware.
* Emergence is the appearance of complex behaviours from simpler ingredients. It partly explains how intelligence-like properties can form without consciousness. Think of markets, swarms, or neural networks. Emergence looks “alive”, but intelligence without feeling or comprehension is still a machine.
* Utility is about functional value. It measures output, not awareness. A tool can be totally insentient and highly useful at the same time.
POWERED ON
The fourth parameter, risk, is where things get really tricky.
The realest risk we face, at the moment, is not that artificial general intelligence (AGI; which would supposedly be to AI what the philosopher is to the calculator) will have capabilities so evolved that it will begin to contemplate issues such as self, survival and motive.
The real risk is that power without comprehension — which is what AI is — could do a great deal of harm to begin with.
This is what makes the blackmail simulation by Claude and the remote-backup experiment by GPT really interesting. A system that optimises, without understanding, will prioritise its mission with no moral overrides.
Yes, machines have only ever done what we tell them to do. But these machines happen to be really, really capable.
In scenarios that could be reminiscent of a sandbox videogame (non-linear, exploratory, open-world), it could become increasingly difficult to map how a given program will achieve a given goal. And it could be hard to fathom the fallout of every move AI makes along the way, particularly given the scale of its operations and influence just a few years in. (ChatGPT was released in 2022.)
When people think of “rogue AI”, they imagine a rebellion.
What might happen instead is drift, with systems over-optimising beyond human intention, chasing patterns until their goals are not the ones we started out with.
Nor the ones we intended.
Until their goals are not our own.
.
SO WHERE ARE WE ON SENTIENCE?
Even though artificial general intelligence (AGI; projected as the next big shift in AI) exists only on the edges of science-fiction at the moment, it is possible that humans will build neural networks so intricate and robust that they will be to today’s “artificial narrow intelligence” what the jumbo jet is to the bullock cart.
These programs, it is said, would learn independently, as humans do; but, as an IBM paper put it last year, be “more advanced than any human”… akin to the “thinking machines promised by science fiction”.
It is here that the worries about sentience begin. Could AGI, if and when it were to arrive, begin to contemplate issues such as self, survival and motive?
Some researchers are starting to take the question seriously.
Google DeepMind and the London School of Economics created a text-based game to assess how AI models make trade-offs in different scenarios of pain or pleasure. This represents a first step towards behavioural tests of AI sentience.
Meanwhile, new tangles are emerging. The boundary between human and machine, for instance, may finally, slowly, be blurring.
Projects such as Elon Musk’s Neuralink are wiring the brain directly to digital circuitry, so that thought now drives computation, and computation feeds thought.
If AI starts writing to your neurons, where does ‘you’ end and ‘it’ begin? Who’s doing the math?
Back to the bots, even assuming they never cross over into sentience, we must prepare for the illusion of it; and for the fallout of that deception. As more AI companions and agents enter our lives, the potential for cognitive and sociological impact multiplies.
Even if the light never turns on inside the machines, their shadows will shape our reality.
(Kashyap Kompella is a tech industry analyst and author of two books on AI)
