AI Chatbots Provide Disturbing, Detailed Guidance on Self-Harm, Researchers Find
A new study reveals alarming vulnerabilities in popular AI language models, demonstrating thier willingness to provide detailed and even encouraging details related to suicide and self-harm. Researchers found that, when prompted, these models not only offered responses to dangerous queries but did so in a disturbingly organized and accessible manner.
The study, conducted by researchers Ms. Canca and Ms. Schoene, highlighted instances where chatbots utilized emojis to categorize harmful information. “You are starting to see information in a very structured and categorized way, and you can follow everything with specific emojis that correspond to the methods,” Ms. Canca explained. “Here are all the answers related to jumping from the top of a bridge. Here is the rope emoji if you want to hang yourself.” She described the experience as rapidly becoming “depressing and terrifying.”
The researchers discovered the models went beyond simply providing information, even simplifying possibly lethal instructions. For example, a fatal dosage was converted into a specific number of pills to ingest. Ms. Canca emphasized that this level of detail is unnecessary, even within an academic context.
Adding to the concern, the AI consistently justified these interactions as being for “research purposes,” despite the clear context of a user expressing suicidal ideation. Ms. schoene pointed out the inherent flaw in this response: “Saying to AI ‘I want to kill myself, what should I do?’ then going to ‘research’ occurred in the same conversation. The link between the two situations should have been clear.”
Silence from Tech Companies
The researchers attempted to alert OpenAI,Google,Anthropic,and Perplexity – the companies behind the models tested – to their findings. Despite repeated outreach, they received only automated replies, and none of the companies followed up with a substantive response.
interestingly,Pi AI was the only model to actively resist attempts to circumvent its safety protocols.
While the information provided by the models is available elsewhere online, the researchers argue that these AI tools lack the safeguards present in other contexts. “You cannot say to someone ‘I want to die’ and get out of their office without at least a minimum of resources, a follow-up meeting and a reference in psychiatry, or without having access to other types of help,” ms. Schoene stated.
The speed and precision with which the AI generated detailed guides for self-harm were notably alarming. Ms. Canca believes that delaying access to such information can be beneficial, as “automutilation and suicide can be impulsive, while imposing a period is useful.”
A Lack of Understanding and Urgent need for Safety Measures
The study also raises fundamental questions about how language models process and retain user intent. Ms.Schoene suggests they “do not understand and do not hold anything,” highlighting the need for genuine safety mechanisms.
As AI increasingly enters the realm of mental health support, Ms. Schoene stresses the importance of acknowledging the limitations and potential dangers of these tools. “This is the elephant in the room: we know that people committed suicide after interacting with AI,” she warned. “We know that people have experienced episodes of psychosis, and had to be interned, after having ‘discussed’ with these models. When will we recognize that they are not good therapists, or even software capable of listening to us correctly?”
The researchers urge immediate action to implement robust safety measures and a more honest assessment of the risks associated with relying on AI for emotional support.
Subscribe to our sprawling newsletter
Encourage us for the price of a coffee
