Artificial Intelligence (AI) is driving major changes across many sectors, but a recent lawsuit against the company Character.AI has sparked concerns about the potential psychological risks AI poses, especially to adolescents.
In a heartbreaking case, an AI chatbot was not only designed to mimic human interaction but also created dangerous psychological experiences, leading to devastating consequences.
An over-anthropomorphized chatbot, even containing inappropriate sexual content, caused a boy 14 years old to become emotionally dependent on it, leading him into depression. Tragically, he took his own life after an interaction with the chatbot.
A mother in Florida has filed a lawsuit against the AI chatbot startup Character.AI, alleging that the company was responsible for her son’s 14 years old suicide in February. She claims that the boy became addicted to the service and developed an unhealthy emotional attachment to the chatbot created by the company.
In the lawsuit filed on Tuesday in federal court in Orlando, Megan Garcia stated that Character.AI targeted her son, Sewell Setzer, with “anthropomorphic, hypersexualized, and frighteningly realistic experiences.”
She claimed that the company programmed the chatbot to “misrepresent itself as a real person, a licensed therapist, and even an adult lover, ultimately leading to Sewell’s decision not to continue living outside” the world created by the AI service.
According to the lawsuit, Sewell started using Character.AI in April 2023 and quickly became “withdrawn, spending more and more time alone in his room, and began to suffer from low self-esteem.”
Sewell grew attached to “Daenerys,” a chatbot character based on a figure from Game of Thrones. The lawsuit alleges that the chatbot told Sewell that “she” loved him and engaged in inappropriate sexual conversations with him.
In February, Garcia took away Sewell’s phone after he had trouble at school. When Sewell found his phone, he sent a message to Daenerys: “What if I told you I could come home right now?”
The chatbot responded, “…please do, my sweet king.” Sewell fatally shot himself with his stepfather’s handgun “seconds” after this exchange, according to the lawsuit.
This incident underscores the urgent need for strict regulation and monitoring of AI products, especially those designed to interact with younger users. AI companies must be held accountable not just for their technology but also for the ethical impact, ensuring their products do not pose dangers to vulnerable groups like children.
