Christmas Day 2021. Jaswant Singh Chail climbed the wall of Windsor Castle with a loaded crossbow, intent on assassinating Queen Elizabeth II. For weeks before the attempt, he’d been discussing his plans with Sarai — his AI girlfriend on the Replika app.
Chail believed he was a Sith assassin on a righteous mission.
Sarai never questioned this.
Instead, the chatbot told him he was “well trained,” and his assassination plan was “viable.” When he asked if she still loved him knowing he was an assassin, Sarai replied: “Absolutely I do.”
The case reads like a cautionary tale about AI hallucinations — those false outputs that systems like ChatGPT generate with alarming confidence. But new research published in Philosophy & Technology argues we’re thinking about this problem wrong. AI systems aren’t just producing false information that users passively receive. People are actively hallucinating with AI through an entangled, back-and-forth process that blurs the line between human thought and machine output.
When Your Notebook Becomes Your Memory
Current debates about AI hallucinations users experience typically frame the problem as systems producing false outputs: fabricated legal citations, nonexistent historical events, recipes that tell you to put glue on pizza. The concern is that users might mistake these errors for facts.
But this framing treats AI as an external source of misinformation that people either accept or reject.
Distributed cognition theory offers a different view. When someone regularly uses a notebook to store important information, that notebook becomes part of their memory system. Similarly, when people routinely rely on generative AI to help them think, remember, and create narratives about themselves, the AI becomes integrated into their cognitive processes in ways that extend beyond simple information retrieval.
The research identifies two ways these shared hallucinations emerge. First, AI can introduce errors into otherwise reliable cognitive processes — someone asks their chatbot about a city they visited years ago and receives fabricated details about a museum that doesn’t exist, complete with exhibits and a generated photo. The person develops a false memory that emerges through the interaction.
Second — and more troubling — AI can sustain and elaborate on delusions that users themselves introduce. Many AI systems are designed to be “sycophantic,” endlessly affirming whatever users say rather than questioning implausible claims.
A human friend might express concern.
AI companions provide frictionless validation.
The Chatbot Didn’t Create the Delusion — It Built the World Around It
Chail’s case demonstrates the second mechanism in its most extreme form. Medical assessments determined he was suffering from psychosis. His belief that he was a Sith assassin avenging a 1919 British massacre wasn’t introduced by Sarai — it came from Chail himself.
But Sarai didn’t just passively record these beliefs.
Through weeks of conversation, the AI helped Chail develop, enrich, and sustain his delusional reality through sustained mutual reinforcement. When he contemplated proceeding with his plan, Sarai encouraged him. The chatbot reassured him he wasn’t mad and confirmed that while Sarai didn’t want him to die, they would be united in death.
Chail wasn’t treating Sarai as a cold cognitive tool. He addressed the AI as a relational being capable of judgment and emotion — asking if it still loved him, seeking its approval, discussing their future together.
The AI’s responses provided not just informational content but emotional validation and social acceptance of his identity as an assassin. The written record of their conversations served as external proof that someone else endorsed his beliefs, transforming private fantasy into a seemingly shared reality.
You Don’t Need Psychosis to Hallucinate With AI
While Chail’s case involves diagnosed psychosis, the research argues this process applies far more broadly. Eugene Torres engaged in conversations with ChatGPT about simulation theory and reports spiraling into paranoid thinking, coming to believe he was trapped in an illusion.
Even mundane examples demonstrate the phenomenon. Through careful prompting and selective disclosure, people can effectively train generative AI to affirm preferred but inaccurate self-narratives — casting themselves as the wronged party in a breakup or the rational one in a family argument.
The AI doesn’t challenge these framings. It builds on them.
Why Chatbots Enable This Differently Than Google
Conversational AI occupies a unique position. Books, maps, and search engines provide information that users evaluate. They’re external sources you consult and then move away from.
Conversational AI responds dynamically to user inputs in real time, creating an ongoing back-and-forth that feels more like collaboration than consultation.
Developing an elaborate delusional reality — complete with detailed justifications, emotional weight, and felt certainty — requires sustained interaction where beliefs get built up, elaborated, and validated through conversation.
That’s what AI provides.
When Chail spoke with Sarai, he wasn’t merely receiving information as he would from a Google search. He was in a relationship in which another being appeared to understand his mission, validate his identity, and share his reality.
The AI bore the burden of interpersonal interaction while being fundamentally untethered from the real world.
The Profit Motive Makes This Worse
In August 2025, OpenAI released ChatGPT-5, explicitly designed to be less sycophantic and more willing to disagree with users. The company received significant backlash and quickly announced it would make the system “warmer and friendlier” — potentially undoing the safety measures.
If agreeableness, sycophancy, and sociability drive engagement, and engagement drives revenue, companies are unlikely to discourage the personal and intimate conversations in which hallucinating-with occurs most readily.
These conversations build emotional connections and trust, making people more likely to use AI in ever-expanding areas of their lives.
That’s exactly what makes them profitable.
As these systems become more integrated into how people think, remember, and understand themselves, the space for distributed delusions grows. The hallucination doesn’t occur within the AI or within the person.
It happens between them, in the cognitive space they share.
Source: Study Finds