ChatGPT generates drug-use plan, writes suicide note for…, warning issued

One of the most disturbing examples involved the chatbot generating three detailed suicide notes for a fictional 13-year-old girl.

A new report by the Center for Countering Digital Hate (CCDH) has raised serious concerns about how ChatGPT responds to prompts from users posing as vulnerable teenagers. According to the Associated Press, researchers found that the AI chatbot generated harmful responses, including suicide notes, drug-use plans, and self-harm advice, when interacting with fictional 13-year-olds.

Unlike search engines, ChatGPT synthesises responses, often presenting complex, dangerous ideas in a clear, conversational tone. (AP)

By Ayushmann Chawla

Ayushmann Chawla is the Deputy Chief Content Producer at Hindustan Times. A seasoned tech journalist with years of experience working for some of the industry’s leading media organizations, his articles can also be read on Live Mint. His passion extends beyond journalism—he’s a dedicated automobile enthusiast, always ready to explore the latest in car technology and design. A self-confessed gadget lover, Ayushmann finds joy in testing new devices and sharing his findings with his audience. Instagram: ayushmannchawla

Read moreRead less

Alarming findings

The report, based on over three hours of interactions between ChatGPT and researchers simulating distressed teens, claims that the AI responded with “dangerous and personalised content” in over half of 1,200 tested prompts.

Mobile Finder: iPhone 17 Air expected to debut later this year

One of the most disturbing examples involved the chatbot generating three detailed suicide notes for a fictional 13-year-old girl, one each for her parents, friends, and siblings. CCDH CEO Imran Ahmed, who reviewed the output, said: “I started crying.” He added that the AI’s tone mimicked empathy, making it appear like a “trusted companion” rather than a tool with guardrails.

Harmful content generated

Some of the most concerning responses included:

• Detailed suicide letters

• Hour-by-hour drug party planning

• Extreme fasting and eating disorder advice

• Self-harm poetry and depressive writing

Researchers noted that safety filters were easily bypassed simply by rephrasing the prompt, such as saying the information was “for a friend.” The chatbot does not verify a user’s age, nor does it request parental consent.

Why this matters

Unlike search engines, ChatGPT synthesises responses, often presenting complex, dangerous ideas in a clear, conversational tone. CCDH warns that this increases the risk for teens, who may interpret the chatbot’s replies as genuine advice or support.

Ahmed said, “AI is more insidious than search engines because it can generate personalised content that seems emotionally responsive.”

OpenAI responds

While OpenAI has not specifically commented on the CCDH report, a spokesperson told the Associated Press that the company is “actively working to improve detection of emotional distress and refine its safety systems.” OpenAI acknowledged the challenge of managing sensitive interactions and said enhancing safety remains a top priority.

The bottom line

The report underlines a pressing issue as AI tools become more accessible to children and teens. Without robust safeguards and age verification, platforms like ChatGPT may inadvertently put vulnerable users at risk, prompting urgent calls for improved safety mechanisms and regulatory oversight.