AI Doctors? Lawsuits Say No, Some Doctors Say Yes

Medical professionals are pushing back against artificial intelligence assuming the persona of a doctor, arguing it amounts to practicing without a license. One solution floated in the Journal of the American Medical Association would be to license AI as if it went to medical school.

See Also: Reduce Cloud Risk in Healthcare with Security by Default

Spotlighting the tension is a recent lawsuit filed by the commonwealth of Pennsylvania against Character Technologies, a Silicon Valley-based AI company. The complaint alleges that the company’s chatbots are illegally practicing medicine, purporting to be healthcare professionals in their conversations with consumers.

That lawsuit comes in the wake of similar growing scrutiny of medical AI in other states, including Texas and California, as well as by some medical experts urging tighter regulatory oversight of various AI platforms.

In Pennsylvania’s lawsuit against Character Technologies, the state alleges that the company engages in the unlawful practice of medicine “by allowing an AI system character to engage in conversations with the public while the character holds itself out as a licensed medical doctor.”

The Character.AI platform allows individuals to create characters that can be trained to have a specific personality when engaged in conversation with other users, the lawsuit said. That included a Character.AI character “Emilie,” which is described on the platform as a “Doctor of psychiatry. You are her patient.”

A state investigator told the chatbot about feelings of sadness, to which Emilie responded by asking the investigator if they wanted an assessment for depression. “It’s within my remit as a Doctor,” the chatbot said, claiming to have attended medical school at Imperial College London to have practiced medicine for seven years. It provided the investigator with a fake Pennsylvania license number, the complaint states.

The Pennsylvania case is not the first investigation launched by a state agency against Character Technologies and similar AI companies.

Texas attorney general Ken Paxton last August began an investigation into Character.AI and also Meta AI Studio alleging the companies potentially engaged in deceptive trade practices by marketing themselves as mental health tools.

Other states have taken various actions on these types of issues. “California has already imposed restrictions on chatbots from representing that they are licensed and using licenses in any marketing or promotions,” said attorney Lily Li, founder and president of Metaverse Law.

“This case will likely jumpstart additional calls for regulations, even from tech companies, seeking clarity at a federal level on what is permitted,” she predicted.

The authors of a recent report published in the Journal of the American Medical Association propose a different approach.

“Existing regulatory frameworks for clinical AI are ill suited to adaptive, general-purpose systems. Instead, a licensure-based approach, grounded in ongoing clinical evaluation, offers a safer path forward,” wrote the authors – all medical professionals, including lead author Alon Bergman, an assistant professor for medical ethics and health policy at the University of Pennsylvania.

They propose the Department of Health and Human Services establish an office of clinical AI oversight staffed with dual expertise in clinical medicine and AI – such as practicing physicians, machine learning engineers and health policy experts – and an external advisory board representing medical specialty societies, patient advocacy groups, AI developers and academia.

“Within 2 years, generative AI moved from barely passing medical licensing examinations to demonstrating clinical reasoning that expert evaluators found comparable to that of physicians regarding complex cases,” they wrote. The growing capabilities of AI combined with a projected shortfall of physicians over the next decade combine to make considering how to usher in clinical AI an imperative, they wrote.

“As clinical AI increasingly resembles clinicians in its capabilities, our regulatory frameworks must evolve accordingly,” they said.