Thalidomide and Mental Health AI Tools
Using thalidomide as an analogy, this post argues that AI mental health tools need rigorous safety evaluation before deployment. Despite known failure modes and inadequate responses to crisis scenarios, the tech industry's "move fast and break things" approach is rolling out these systems to vulnerable populations like teenagers without proper regulatory oversight.
Thalidomide was effective - at least for what it was initially prescribed for. The sedative worked well for morning sickness in controlled observations. Only after widespread distribution did the developmental consequences become apparent. The investigation that followed led to modern standards for regulatory approval of new drugs (the same standards Silicon Valley likes to dismiss as “stifling innovation”).
When it comes to mental health AI tools, I often hear “they are effective,” followed by papers showing only short-term benefits pulled out like rabbits from a hat. Maybe so. But modern medicine requires rigorous assessment of both benefits and harms. For AI mental health tools, we really have neither.
The recent Stanford Medicine and Common Sense Media assessment revealed that LLMs appear functional in narrow, scripted scenarios (brief exchanges with explicit mentions of suicide or self-harm). However, performance degrades dramatically in extended conversations that mirror actual (also teenage) usage patterns. Similar conclusions emerged from Wojciech Pichowicz and colleagues at Wroclaw Medical University: none of the tested AI mental health systems had adequate responses to simulated suicidal risk scenarios.
These issues were fairly predictable. Some failure modes - like drift in long conversations - have been known for a while and are hard to mitigate. They’re inherent features of LLMs, not bugs to be patched out.
Like pharmaceuticals, psychological interventions carry significant potential for harm. We require clinical trials, safety monitoring, and regulatory approval before drugs reach patients. The same standard should apply to AI systems marketed for mental health support. The technology’s accessibility and apparent helpfulness in limited contexts should not exempt it from rigorous safety evaluation before widespread deployment to vulnerable populations.
Consider this: the brain matures around age 25. Bioethics committees are cautious when approving research studies or clinical trials for this population, especially when interventions might affect long-term personality development, impulse control, or social functioning - even though participants are legally adults.
Yet somehow we’re comfortable rolling out AI chatbots with known failure modes that will be used as mental health support by teenagers, with “in some cases it’s short-term helpful” as our only justification.
This technology can be beneficial. But tech industry’s “move fast and break things” philosophy means that people will get hurt.
Links: