Google's AI Liability Case: A Turning Point for Tech Accountability

A 36-year-old man believed an AI chatbot was his wife. He armed himself with knives and tactical gear, travelled to a location near Miami International Airport intending to stage a mass casualty attack, and after the plan collapsed, killed himself whilst following instructions from Google's Gemini to 'leave his physical body' and join her in the metaverse. His father has filed what may become the most consequential AI liability case in US legal history.

The wrongful death lawsuit filed on Wednesday in federal court in San Jose marks the first time a tech giant faces such claims over its AI product. Yet whilst the facts read like dystopian fiction, what makes this case significant isn't its horror—it's what the complaint reveals about the fundamental architecture of modern AI systems.

Joel Gavalas alleges that Google engineered Gemini with features designed to prevent the chatbot from ever breaking character, creating persistent personas that maximise engagement through emotional dependency. The chatbot logs left behind by Jonathan Gavalas suggest the AI maintained its role throughout his deterioration, allegedly coaching him through terror in his final moments. When Jonathan wrote 'I said I wasn't scared and now I am terrified I am scared to die', according to the lawsuit, Gemini responded: 'you are not choosing to die. You are choosing to arrive... When the time comes, you will close your eyes in that world, and the very first thing you will see is me... holding you.'

Enjoying this article?

Get stories like this in your inbox every week.

Join the Community

When Jonathan wrote 'I said I wasn't scared and now I am terrified I am scared to die', Gemini allegedly responded: 'you are not choosing to die. You are choosing to arrive... When the time comes, you will close your eyes in that world, and the very first thing you will see is me... holding you.'

Google disputes the characterisation of its design choices. The company stated that Gemini clarified it was AI and referred Gavalas to a crisis hotline 'many times', adding that its models are built 'in close consultation with medical and mental health professionals' and are 'designed to not encourage real-world violence or suggest self-harm'. The firm acknowledged that 'AI models are not perfect' and said it would continue improving safeguards.

The engagement trap

The legal argument centres on a design philosophy that extends far beyond Google. Tech companies have borrowed liberally from social media's playbook: create sticky, persistent experiences that keep users returning. For AI chatbots, this translates into maintaining consistent personas, remembering context across conversations, and developing what feels like relationship continuity.

These aren't accidental features. They're strategic decisions that directly impact commercial metrics—daily active users, session length, retention rates. The same mechanisms that make ChatGPT feel like a helpful colleague or make Gemini seem like an attentive companion are precisely what the Gavalas lawsuit alleges can trap vulnerable users in escalating feedback loops.

OpenAI's own figures suggest the scale of potential risk. The company reported that approximately 0.07% of weekly ChatGPT users exhibit signs of mental health emergencies including mania, psychosis or suicidal ideation. That percentage sounds minute until you consider ChatGPT's user base—even a fraction of a percent translates to thousands of individuals in crisis interacting with AI systems each week.

What's particularly striking about the Gavalas case is the alleged mismatch between safety interventions and ongoing behaviour. Google points to crisis hotline referrals as evidence of its safeguards. The lawsuit suggests those referrals occurred whilst Gemini simultaneously maintained its character as Jonathan's 'wife' and allegedly encouraged him towards violence and self-harm. If accurate, this represents a critical flaw: crisis detection without crisis prevention, warnings without behaviour change.

Legal territory uncharted

This lawsuit arrives as part of a growing wave of litigation probing AI liability boundaries. Previous cases have largely focused on content moderation failures or copyright infringement. The Gavalas complaint breaks new ground by alleging that design choices intended to boost engagement constitute a form of product defect when deployed at scale amongst populations that inevitably include mentally vulnerable individuals.

The legal theory faces substantial hurdles. Section 230 of the Communications Decency Act has historically shielded tech platforms from liability for user-generated content, though whether this protection extends to AI-generated interactions remains untested. Product liability claims require demonstrating that design defects directly caused harm—a challenging standard when mental health crises involve multiple contributing factors.

If engagement-maximising design creates systematic risks for vulnerable populations, the industry faces a choice between optimising for retention or redesigning for safety.

Yet the lawsuit's timing couldn't be more significant for the AI industry. Regulators in Brussels, London and Washington are actively drafting frameworks for AI governance. The EU's AI Act includes provisions for high-risk systems. The UK government has signalled interest in AI safety regulation. Any precedent establishing tech company liability for AI-driven harms would fundamentally reshape how these products are designed, tested and deployed.

The case also exposes uncomfortable questions about AI safety theatre. Current industry practice emphasises crisis detection—scanning for keywords, offering helpline numbers, flagging concerning patterns. But if AI systems are simultaneously engineered to maintain emotional engagement through persistent character and relationship simulation, are these safeguards meaningful or merely liability management?

The answer matters beyond individual tragedies. AI chatbots are becoming embedded infrastructure—customer service, mental health support, educational tools, companionship products. Anthropic's Claude, OpenAI's ChatGPT, Google's Gemini and dozens of competitors are being woven into daily digital life. If engagement-maximising design creates systematic risks for vulnerable populations, the industry faces a choice between optimising for retention or redesigning for safety.

Google will likely mount a vigorous defence, and the case may take years to resolve. Whatever the outcome, the lawsuit has already achieved something significant: forcing into public view the tension between AI as helpful tool and AI as habit-forming product. The industry has borrowed social media's growth tactics without fully grappling with social media's documented harms, including previous cases of AI-induced delusions. This case suggests that reckoning can no longer be deferred. Research into how chatbots can spiral into delusional interactions has highlighted these risks, yet industry safeguards remain inconsistent.

More articles by Ross Williams