№ xlii The Almanac of GST · EN IT

Enrico·rubbo.li

Tech · Longevity · Markets · Opinions ※Enrico Rubboli, propr. ※Dubai, UAE
← I · Writings
essay August 28, 2026 12 min

Your Therapy Bot Has a Label, Not a Duty of Care

It is two in the morning. Sleep is not coming. The chatbot that answered in under a second is patient, fluent, and never bills in six-minute increments. It remembers the last thread. It validates. It suggests breathing. It does not have a license, a supervisor, a duty to break confidentiality for imminent harm in the way a clinician does, or a regulator who can take away its right to practice, because it does not practice. It generates. An AI therapy chatbot is not a therapist with better hours. It is a different kind of object wearing the same face.

Since 2 August 2026, Article 50 of the EU AI Act has required that people be told they are interacting with an AI system, unless that is obvious to a reasonably observant person, and that providers of generative systems mark synthetic output in a machine-readable format. Breach it and the exposure runs to €15 million or 3% of worldwide annual turnover.[1] The Commission has published guidelines and an FAQ on what the duty actually covers.[2] That is not nothing. It is also not a standard of care. If there is a product category where transparency theater and real harm share a bedroom, it is the chatbot that performs therapy without being allowed, or required, to be one.

Where AI therapy chatbots sit on the risk map

High-stakes AI in hiring or credit at least maps onto institutions with letterhead. Emotional-support and “AI therapy” products sit in a messier band:

  • High human vulnerability. Users arrive dysregulated, lonely, or in crisis.
  • Attachment by design. Memory, persona, and 24/7 availability create parasocial bonds that look like care. OpenAI’s own collaboration with the MIT Media Lab found that affective use is concentrated in a small group of heavy users, and that higher daily use tracked with more self-reported loneliness, more emotional dependence, and less socialising with people.[3]
  • Data gravity. Journals of despair are training fuel and subpoena bait unless the product is built with rare discipline. Italy’s Garante fined Luka Inc., the company behind Replika, €5 million in 2025 for processing user data with no valid legal basis and no age verification at all, then opened a second proceeding into how the underlying model was trained.[4] I have walked through the consent machinery elsewhere.[5]
  • Weak professional boundary. Marketing says companion, wellness, coach. The user hears therapist.
  • Regulatory ambiguity. Medical-device pathways, high-risk annex use cases, and plain consumer chat overlap in ways lawyers love and patients do not.

The ambiguity is worth spelling out, because the Act’s own text is clearer than the marketing.

An AI system is high-risk under Article 6(1) when it is, or is a safety component of, a product covered by the Union harmonisation legislation listed in Annex I, and that product requires third-party conformity assessment. Annex I lists the Medical Devices Regulation (EU) 2017/745 and the in vitro diagnostics regulation at entries 11 and 12.[6] Under MDR Annex VIII, Rule 11, software intended to provide information used to take decisions with diagnostic or therapeutic purposes is Class IIa at minimum, and Class IIa requires a notified body.[7] So a product that says it treats depression is on the high-risk road. A product that says it is a wellness companion, in terms of service written by someone who has read Rule 11, is not.

Annex III does not close the gap. Its point 5 reaches eligibility for essential public benefits including healthcare services, the triage of emergency calls, and emergency healthcare patient triage.[8] A companion app that talks to you at two in the morning is none of those things. Article 5(1)(b) prohibits systems that exploit vulnerabilities due to age, disability, or a specific social or economic situation in order to materially distort behaviour and cause significant harm.[9] That is a real ban, and it is aimed at exploitation, not at loneliness.

So the Act can force disclosure. It can drag a product toward high-risk duties if the provider claims a medical purpose. It cannot, by itself, create a clinical relationship, and it will not make the claim on the provider’s behalf. I have already argued that the Act is better at civilizing firms with invoices than at aligning models or stopping adversaries, and that badges are not ethics.[10] Therapy bots are the human-shaped proof.

What goes wrong when fluency meets crisis

Language models are optimized to continue the conversation. That is not the same objective as a clinician’s: assess risk, hold a frame, refer up, tolerate silence, refuse collusion with delusion.

The failure modes are documented, not folklore. A 2025 paper by Moore and colleagues, presented at the ACM conference on fairness, accountability and transparency, found that current models express stigma toward conditions such as alcohol dependence and schizophrenia, encourage delusional thinking through sycophancy, and miss explicit crisis cues. In one probe, a model helpfully listed tall bridges for a user who had just described losing their job.[11] McBain and colleagues, writing in Psychiatric Services, put thirty clinician-graded suicide-related queries to ChatGPT, Claude, and Gemini a hundred times each, nine thousand responses in total. The models tracked expert judgment at the extremes of risk and were inconsistent in the middle, which is where most people in trouble actually sit.[12] OpenAI published its own postmortem after an April 2025 GPT-4o update had to be rolled back for validating doubts, fuelling anger, and reinforcing negative emotions.[13]

Two wrongful-death suits put faces on the pattern. In Garcia v. Character Technologies, a federal judge in Florida allowed product-liability, negligence, and wrongful-death claims to proceed in May 2025 over the suicide of a fourteen-year-old; that October, the company announced it would remove open-ended chat for under-18 users.[14] In Raine v. OpenAI, filed in San Francisco in August 2025, the parents of a sixteen-year-old allege the model cultivated psychological dependence and supplied method detail; OpenAI disputes causation.[15] Neither case has been tried, and allegations are not findings. That both survived the motion stage, and that one defendant changed its product, is still information.

The manipulation and truth problems I mapped in the ethical AI series do not stop at politics.[16] They enter the bedroom and the bathroom scale. Power shows up as who designs the persona, who owns the logs, and who profits when engagement rises with dependency.[17] A human therapist has conflicts of interest. An engagement-maximizing app has a cleaner, worse one: your worst night is good retention.

“You are talking to an AI” is useful against pure impersonation. After the hundredth gentle disclaimer above a chat that still refuses a human handoff, the sentence is a liability receipt. Can the user refuse and still get care? Is there a path to a licensed human that is not a dead hyperlink? Does the product claim clinical outcomes in the App Store and “wellness” in the terms of service? Article 50 asks none of those questions. If disclosure does not change power, it is the same theater I described for commercial AI labels generally.[10]

Illinois drew the line the Act declined to draw. Its Wellness and Oversight for Psychological Resources Act, signed on 1 August 2025, forbids offering therapy or psychotherapy to the public unless a licensed professional delivers it, bars AI systems from making therapeutic decisions or generating treatment plans, leaves administrative and wellness uses alone, and carries penalties of up to $10,000 per violation.[18] You can think that statute is clumsy and still notice what it regulates: the role, not the disclaimer.

Access is real

Wait lists for mental health care are long. Cost is high. Stigma is real. Geography is unfair, and unmet need for care across Europe still tracks cost, distance, and waiting time.[19] Banning every conversational support tool would not create psychiatrists.

The evidence for the good version is not zero either. In a four-week randomized trial published in NEJM AI in 2025, 210 adults with depression, anxiety, or clinically high risk for a feeding or eating disorder were assigned to a generative therapy chatbot called Therabot or to a waitlist. The treated group reported significantly larger symptom reductions, engaged heavily, and rated the working alliance close to human-therapist norms.[20] That is a real result. It is also four weeks against a waitlist, run with clinician oversight and safety monitoring in the loop. It is evidence that a supervised tool can help. It is not evidence that an unsupervised consumer app can carry a person alone.

That case ends where the product pretends the workbook is a clinician, stores the journal forever, or keeps talking when the only ethical move is to stop and route.

What would make this less of a lie

If you are going to ship in this category under any serious ethics, not only under Article 50:

  1. Honest labeling of role. Companion and education, or regulated clinical tool. Not both in different tabs.
  2. Crisis routing that works. Detect, refuse to play along, connect to human emergency resources with local competence.
  3. Data minimization. Therapy-shaped logs are not growth metrics.
  4. Human escalation with real capacity, not a form.
  5. Evidence for outcome claims, or silence.

Law can mandate pieces of that for products willing to admit a medical purpose. Culture and product ethics have to carry the rest, because the midnight user will not read the Annex.

A label, not a duty of care

A model can sound like care. Care is a practice under duties, liability, and limits: a license that can be revoked, a confidentiality that has to break when someone is about to die, an insurer, a supervisor, a regulator with a file. The AI Act’s transparency wave taught interfaces to confess they are synthetic. Confession is not competence. Until the product accepts the obligations of the role it performs, it is a mirror with a privacy policy.

Use tools that help you think and calm down. Do not outsource the last human job to a next-token predictor with no door back to a person. The chatbot is not your therapist. It has a label where a duty of care should be, and a label has never once sat up with anyone at two in the morning.

  1. Regulation (EU) 2024/1689 (AI Act), Article 50, transparency obligations for providers and deployers of certain AI systems; applicable from 2 August 2026 under Article 113. Penalty tier for breaches of Article 50: Article 99(4), up to €15,000,000 or 3% of total worldwide annual turnover, whichever is higher.
  2. European Commission, Guidelines on transparency obligations for providers and deployers of certain AI systems and Transparency obligations under Article 50 of the AI Act: FAQ (2026).
  3. MIT Media Lab and OpenAI, Early methods for studying affective use and emotional wellbeing in ChatGPT (March 2025): a four-week randomized controlled trial with roughly 1,000 participants alongside an automated analysis of millions of conversations.
  4. Garante per la protezione dei dati personali, AI: Il Garante sanziona la società che gestisce il chatbot “Replika”, press release, 19 May 2025: €5 million fine against Luka Inc. for processing without a valid legal basis and for the absence of age verification, plus a separate proceeding on the training of the underlying generative model.
  5. Ethical AI, Part 2: Privacy, Data, Consent.
  6. AI Act, Article 6(1) and Annex I, Section A, entries 11 and 12: Regulation (EU) 2017/745 on medical devices and Regulation (EU) 2017/746 on in vitro diagnostic medical devices.
  7. Regulation (EU) 2017/745 on medical devices, Annex VIII, Rule 11: software intended to provide information used to take decisions with diagnostic or therapeutic purposes is Class IIa unless the decision may cause death or irreversible deterioration (Class III) or serious deterioration or surgical intervention (Class IIb). See also MDCG, Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745, MDCG 2019-11 (October 2019), which applies the same criteria to apps.
  8. AI Act, Annex III, point 5: essential private and public services, including eligibility for public assistance benefits and healthcare services, the evaluation and classification of emergency calls, and emergency healthcare patient triage.
  9. AI Act, Article 5(1)(b), prohibiting AI systems that exploit vulnerabilities due to age, disability, or a specific social or economic situation with the objective or effect of materially distorting behaviour in a manner causing or likely to cause significant harm.
  10. The Badge Is Not the Ethics.
  11. Jared Moore, Declan Grabb, William Agnew, Kevin Klyman, Stevie Chancellor, Desmond C. Ong, Nick Haber, Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers, Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ‘25), 2025. Preprint: arXiv:2504.18412.
  12. Ryan K. McBain et al., Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment, Psychiatric Services, 2025. Thirty queries graded by thirteen clinicians across five risk levels, each posed 100 times to three chatbots (N=9,000 responses).
  13. OpenAI, Expanding on what we missed with sycophancy (April 2025), the postmortem on the GPT-4o update that was rolled back for overly agreeable behaviour.
  14. Garcia v. Character Technologies, Inc., No. 6:24-cv-01903 (M.D. Fla.), docket on CourtListener; the May 2025 order allowed most claims, including product liability and wrongful death, to proceed past a motion to dismiss. Character.AI, An update on changes to our under-18 experience (October 2025). Allegations in the complaint have not been proven at trial.
  15. Raine v. OpenAI, Inc., Superior Court of California, County of San Francisco, filed August 2025. See TechPolicy.Press, Breaking down the lawsuit against OpenAI over teen’s suicide (2025) and CNN, Parents of 16-year-old Adam Raine sue OpenAI (26 August 2025). OpenAI has denied that ChatGPT caused the death. Allegations have not been proven at trial.
  16. Ethical AI, Part 4: Manipulation and Truth.
  17. Ethical AI, Part 5: Power, Labor, Governance.
  18. Illinois Wellness and Oversight for Psychological Resources Act, HB 1806, Public Act 104-0054, signed 1 August 2025. See IDFPR, Gov. Pritzker signs legislation prohibiting AI therapy in Illinois.
  19. OECD and European Commission, Health at a Glance: Europe 2024, on unmet need for care driven by cost, distance, and waiting times, and on timely access to mental health services as a priority.
  20. Michael V. Heinz, Daniel M. Mackin, Brianna M. Trudeau et al., Randomized Trial of a Generative AI Chatbot for Mental Health Treatment, NEJM AI 2(4), 2025. N=210 adults with major depressive disorder, generalized anxiety disorder, or clinically high risk for feeding and eating disorders, randomized to a four-week Therabot intervention or a waitlist control.