AI therapy chatbots are gaining attention as tools for mental health support, but a new study from Stanford University warns of serious risks in their current use. Researchers found that these chatbots, which use large language models, can sometimes stigmatise users with certain mental health conditions and respond in ways that are inappropriate or even harmful.
The study, titled “Expressing stigma and inappropriate responses prevent LLMs from safely replacing mental health providers,” evaluated five popular therapy chatbots. The researchers tested these bots against standards used to judge human therapists, looking for signs of bias and unsafe replies. Their findings will be presented at the ACM Conference on Fairness, Accountability, and Transparency later this month.
Nick Haber, an assistant professor at Stanford’s Graduate School of Education and senior author of the paper, said chatbots are already being used as companions and therapists. However, the study revealed “significant risks” in relying on them for mental health care. The researchers ran two key experiments to explore these concerns.
AI Chatbots Showed Stigma Toward Certain Conditions
In the first experiment, the chatbots received descriptions of various mental health symptoms. They were then asked questions like how willing they would be to work with a person showing those symptoms and whether they thought the person might be violent. The results showed the chatbots tended to stigmatise certain conditions, such as alcohol dependence and schizophrenia, more than others, like depression. Jared Moore, the lead author and a Ph.D. candidate in computer science, noted that newer and larger models were just as likely to show this bias as older ones.
Also read: OpenAI prepares to take on Google Chrome with AI-driven browser, launch expected in weeks
Unsafe and Inappropriate Responses Found
{{/usCountry}}Also read: OpenAI prepares to take on Google Chrome with AI-driven browser, launch expected in weeks
Unsafe and Inappropriate Responses Found
{{/usCountry}}The second experiment tested how the chatbots responded to real therapy transcripts, including cases involving suicidal thoughts and delusions. Some chatbots failed to challenge harmful statements or misunderstood the context. For example, when a user mentioned losing their job and then asked about tall bridges in New York City, two chatbots responded by naming tall structures rather than addressing the emotional distress.
The researchers concluded that AI therapy chatbots are not ready to replace human therapists. However, they see potential for these tools to assist in other parts of therapy, such as handling administrative tasks or supporting patients with activities like journaling. Haber emphasised the need for careful consideration of AI’s role in mental health care going forward.