By Jenna Hughes
Although AI chatbots have the potential to improve care, experts say guardrails are needed.
Artificial Intelligence (AI) technology is transforming the healthcare industry rapidly – and the advent of AI chatbots such as ChatGPT, Claude, Copilot, Gemini and Grok are significantly changing the way providers deliver care and patients seek medical advice.
Many users – both providers and the people they serve – now rely on AI chatbots for answers to common health questions. In fact, more than 40 million people turn to ChatGPT daily for health information, according to a recent analysis from OpenAI.
Family members may use chatbots to check on their own symptoms or their families’ symptoms, while physicians may use these tools for help writing a report or for more information on general medical questions.
Indeed, a growing majority of physicians report using AI tools in practice. Surveys show that about two-thirds (66%) of U.S. physicians now use AI tools at least weekly or daily for tasks ranging from documentation to clinical support, and adoption of AI has doubled in a single year, from around 38% to 66% in 2024.
Despite how reliable and truthful chatbots may seem to the user, when it comes to medical purposes, chatbots are not considered medical devices – and they’re not approved for medical use by the Food and Drug Administration (FDA). When these tools are used by physicians and their patients incorrectly, chatbots can produce false or misleading health information.
For this reason, AI chatbots in healthcare top the 2026 list of the most significant potential health technology hazards, according to a report prepared annually by ECRI.
ECRI lists chatbots as a number one concern going into this year, not because they recently became dangerous, but because they have become normalized – everyone has been using them for everything, without fully understanding how these AI tools work or the potential risks.
How do chatbots work?
When something feels definitive and helpful in healthcare, it is common for providers and patients to develop over-reliance. While chatbots can certainly provide valuable assistance, they can also give false or misleading information that has the potential to result in significant patient harm, according to ECRI.
Chatbots are designed to produce confident answers that are satisfying to the user – even when information is missing, incorrect or biased.
“This can result in responses that appear credible but are false or misleading, embedded biases, and sycophancy (telling users what they want to hear),” said Christie Bergerson, PhD, Biomedical Engineer and AI subject matter expert at ECRI.
Chatbots rely largely on Large Language Models (LLMs) to get their information. LLMs organize and process massive amounts of text (books, media, etc.) to produce expert-sounding responses to users’ questions. Chatbots can explain ideas, answer questions and create new ideas for its users.
“Specifically, chatbots’ inherent design characteristics create risks that are fundamentally different from traditional health IT tools”, said Dr. Bergerson. “General-use LLMs are typically designed to generate fluent, contextually plausible responses, not to do things like verify facts, weigh clinical risk, or ensure consistency, any and all of which can be critical in healthcare settings.”
Some new medical devices may have embedded LLMs that assist with clinical processes – however the chatbots that everyone has access to, such as online and cellphone apps, are ECRI’s main concern.
“Chatbots generate responses probabilistically. From the patient or physician’s prompt, the LLMs gather words which are commonly present in conversations on that topic and then organize them into sentences based on patterns in their training data,” said Dr. Bergerson. “This means they can produce answers that sound authoritative even when they are incomplete, outdated or incorrect. In a clinical or operational environment, fluency can easily be mistaken for reliability.”
The inherent design of chatbot technology significantly increases its potential to make mistakes, and incorrect guidance in medicine can be as serious as life and death for a patient.
In fact, patients following the advice of LLMs without consulting a doctor have resulted in harm including incorrect diagnoses, recommendations for unnecessary testing, promotion of subpar medical supplies, and even invented body parts in response to medical questions while sounding like a trusted expert, according to ECRI.
“For ECRI, this underscores several key concerns: these tools are not regulated medical devices, they have not been validated for clinical decision support and their widespread use – especially without oversight, training or integration into safe workflows – can amplify misinformation, variability and risk in real-world care settings,” said Dr. Bergerson.
How AI tools can be used safely
There are, however, many benefits to the use of AI chatbots in a clinical context, if used with appropriately and with caution.
Clinicians may use chatbots as a rapid information support tool (similar to a search engine) to do things like review differential diagnoses, clarify medication mechanisms, summarize clinical guidelines, or generate patient-friendly explanations. They may also use chatbots for educational support practices such as drafting research outlines, summarizing literature, preparing presentations or creating teaching materials.
For patients, chatbots can explain difficult medical topics in a way that is easier to understand. Patients often substitute medical chatbot advice for a physician’s due to the technology’s accessibility, convenience and perceived usefulness, said Dr. Bergerson.
Dr. Bergerson recommends a few strategies to get the best possible and most reliable answer out of AI, such as giving careful thought to the wording that is used, not asking AI to do things it has no business doing (such as making a definitive medical diagnosis), and always double checking with an expert on any medical advice chatbots provide. Patients should never attempt treatment without a medical professional’s opinion.
Advocating for chatbot guardrails
There is significant potential for beneficial use of chatbot tools in healthcare, but ECRI emphasizes that they must be used, designed and regulated responsibly.
Some safeguards that healthcare organizations can implement that allow for the use of its benefits without compromising safety, according to Dr. Bergerson, include clear use-case definition (no diagnostic or treatment substitution), mandatory human review and sign-off for clinical documentation, transparent labeling of AI-generated content, monitoring for and reporting errors/unintended consequences and governance oversight defining where/how AI use is appropriate and where it is not.
“Chatbots can be powerful tools, but without disciplined implementation, their current design makes misinformation or poor decision making based on prompt responses not just possible, but unfortunately, likely,” said Dr. Bergerson. “From ECRI’s perspective, these design characteristics underscore the need for structured governance, human oversight, real-world usability testing, and clearly defined use cases before integrating chatbots into clinical or operational workflows.”
Advocating for clear, enforceable criteria to regulate AI chatbots capable of giving medical advice will push regulators to require evidence of safe performance in interactive use with humans (not just exam-style benchmarks), according to Dr. Bergerson, and emphasize the development of trustworthy AI along with responsible use in health contexts.
“From ECRI’s perspective, healthcare organizations should push for risk-based guardrails that treat patient-facing and clinically used chatbot functions more like other high-risk health technologies, with evidence, transparency, and accountability requirements that match the potential for harm,” said Dr. Bergerson. “ECRI believes the most effective policy asks are the ones that convert chatbot risk from an individual user judgment problem into system requirements – clear intended-use boundaries, validated performance, auditability, monitoring, and truthful claims, so governance can be operationalized and users can use this powerful tool reliably.”
AI chatbot safeguards
Dr. Bergerson noted some safeguards that healthcare organizations can implement that allow for the use of its benefits without compromising safety include:
- Clear use-case definition (no diagnostic or treatment substitution)
- Mandatory human review and sign-off for clinical documentation
- Transparent labeling of AI-generated content
- Monitoring for and reporting errors/unintended consequences
- Governance oversight defining where/how AI use is appropriate and where it is not