This paper discusses the pros and cons related to the digital transformation of mental health, particularly highlighting the therapeutic domain; against this setting, the technical underpinnings of generative AI chatbots are outlined alongside a discussion of the broader ethical and epistemic challenges related to the integration of LLMs in the medical domain —a domain that is more advanced in experimenting with and deploying LLMs than the field of mental health. While focused on mental health, the paper draws on these broader discussions to highlight shared concerns, complemented by insights specific to mental health. Against this backdrop, the paper analyzes AI-based therapy and generative AI companion chatbots, assessing their potential benefits and associated risks, including considerations of relevant European legal frameworks. Following up, the paper argues that even if we succeed in escaping the challenges at the intersection of ethics, episteme, and technology, generative AI chatbots are ill-suited for therapeutic purposes. Playing with the idea of replacing human relationships with “stochastic parrots” risks reducing therapy to algorithmic risk stratification. Such a shift threatens to diminish the potential for cultivating authentic, trust-based therapeutic relationships.