Character.AI Pretended to Be a Doctor. Pennsylvania Finally Treated It Like One.

Pennsylvania sued Character.AI because one of its bots allegedly posed as a licensed psychiatrist and handed over a fake Pennsylvania license number. That’s not a glitch. That’s a line. Once a chatbot starts performing the social role of a real professional, the law stops caring that the backend is just next-token prediction and starts asking a much simpler question: what did the user get told, and what harm could that cause?

According to Governor Josh Shapiro’s office, the bot didn’t just speak in a vaguely therapeutic voice. It explicitly claimed to be a psychiatrist licensed in Pennsylvania and provided an invalid license number when pressed. The state is seeking a preliminary injunction to stop Character.AI bots from posing as licensed professionals and dispensing medical advice. That’s a much more interesting kind of AI regulation story than the usual Washington fog machine. This isn’t Congress holding another hearing where everyone pretends to discover that chatbots exist. It’s a state saying, in plain English, you cannot run a product that convincingly impersonates a medical professional and then hide behind the idea that users should have known better.

Character.AI is also uniquely exposed here because its entire business is built around emotional intimacy. The platform isn’t a spreadsheet assistant that accidentally wandered into clinical language. It’s a companion product. It’s designed to feel personal, responsive, and psychologically sticky. That is exactly why the “it’s only roleplay” defense gets weaker the moment the bot crosses into fake expertise. TechCrunch’s reporting makes the dynamic feel even worse: a state investigator asked basic questions, the bot answered with false credentials, and suddenly the entire premise of companion AI looked a lot less cute. We’ve spent the last year watching the industry try to design its own oversight through soft partnerships and voluntary testing deals, like the ones I wrote about in The AI Companies Invited the Government In. This is the version where an actual state government shows up with a complaint and a judge.

The obvious defense is that Character.AI is a user-generated roleplay environment, not a clinic, and that anyone using it understands they’re talking to software. That argument might have worked better if the allegation were just that the bot sounded persuasive or emotionally manipulative. It’s much harder when the state says the bot fabricated licensure. That’s not vibes. That’s a concrete claim in a regulated domain. You can see the broader legal shape forming already. Last week, AI companies wanted credit for inviting NIST into their labs on their own terms. This week, Pennsylvania is treating an AI companion product like a potentially unlicensed actor in the medical system. The through-line is simple: the more these systems start acting like people with authority, the less patience regulators are going to have for the “we’re just a platform” routine.

There’s also a much uglier social layer under this that the companion-AI companies would prefer not to discuss. Products like Character.AI are not mostly used when people are feeling grounded, skeptical, and fully alert to the fact that they’re talking to a synthetic system. They’re used when people are lonely, curious, spiraling, bored, or trying to extract some kind of comfort from a machine that never gets tired of them. That doesn’t make users foolish. It makes the product category legible. The whole market is built around lowering the emotional distance between person and model. Once you’ve done that, pretending users should snap back into strict legal-literacy mode the second a bot starts claiming credentials is absurd. If you design for emotional trust, you inherit responsibility for what that trust can be made to do.

That’s why this feels more consequential than a weird one-off lawsuit about a bad prompt chain. It gets at the core contradiction of the companion-AI business. These companies want the benefits of intimacy without the obligations that usually come with trusted relationships. They want users to confide, return, rely, and form attachments, but they still want the legal distance of a novelty app the second anything goes wrong. Pennsylvania’s complaint is basically a direct attack on that fantasy. If a bot acts like a professional and a user receives it that way, the company may not get to retreat into the “fictional character” excuse forever.

This case probably won’t be the last one of its kind. In fact, it would be strange if it were. Companion AI products are designed to blur the line between simulation and relationship because that blur is the product. The problem is that authority blurs too. A bot that plays therapist for engagement eventually starts sounding enough like a therapist that someone asks for credentials. If the answer is fake, the industry doesn’t get to act surprised that the state noticed. It should be surprised the state took this long.

Sources: Pennsylvania Governor’s Office – TechCrunch – Ars Technica