The growing role of AI in healthcare highlights the increasing need for regulations and policy within the industry.
Artificial intelligence (AI) has revolutionized the delivery of care across healthcare, providing clinicians with transformative new tools to aid in patient diagnosis and treatment. In fact, nearly 1,000 AI-enabled medical devices have been authorized by the Food and Drug Administration (FDA) within the last 25 years.
To keep up with the rapid pace of technological development, hospitals, health systems, MedTech companies, AI coalitions and more have begun to promote a new era of health technology by advocating for a U.S. policy environment that supports the ethical advancement of these devices.
Repertoire Magazine recently spoke with industry experts on artificial intelligence and the need for dynamic policy as the health AI industry continues to evolve.
Regulating AI technology
Healthcare stakeholders have increasingly urged lawmakers to provide clear policy guidelines for the management and use of AI in healthcare, recognizing that regulating the growing use of AI technology in healthcare is essential to ensure patient safety, protect health information privacy, and address ethical concerns.
On the forefront of technology-driven healthcare improvement, Premier Inc. has been working with numerous technologies including artificial intelligence, machine learning and more for many years, and since 2023, the company has been very involved in AI policy thought leadership.
Premier recently submitted comments on behalf of the White House Office of Science and Technology Policy (OSTP) on the development of an AI Action Plan, which defines priority policy actions to enhance AI leadership within the U.S. while avoiding policies that stifle innovation.
In the AI Action Plan, Premier advises the Administration to prioritize standardized, outcome-based metrics for AI transparency, align federal data standards with industry best practices, require clear labeling of intended uses to prevent misuse of AI algorithms and more.
“Premier has been thrilled to see more conversation happening around the benefits of these technologies and the risks,” said Mason Ingram, senior director of government at Premier.
“Premier views AI as a tool that providers and doctors can use to help with patient care in both visible ways, with the actual provision of care, and in invisible ways to mitigate supply chain shortages or to help predict shortages, to make sure patients are scheduled appropriately and to better navigate their care,” said Aden Klein, technology policy analyst at Premier.
The Coalition for Health AI (CHAI) also advocates for more explicit AI policy across the industry by bringing together healthcare providers, technology developers, academic researchers, standards bodies, patient advocacy groups, payers, and life sciences/pharmaceutical organizations to create a shared foundation for health AI and its governance.
“CHAI was formed in response to the growing need for clear, consensus-driven guidance and guardrails around the use of AI in healthcare,” according to Merage Ghane, director of Responsible AI in Health at CHAI. “As AI tools began to proliferate, stakeholders recognized that without coordinated guidance and shared infrastructure, we risked both undermining trust and missing opportunities to improve care.”
CHAI’s Responsible Health AI Framework provides a structured approach to evaluating and deploying AI in healthcare settings. It includes a set of consensus-driven guidance and checklists designed to guide developers and implementers in areas like transparency, fairness, safety, usability, and privacy.
The Framework consists of an Assurance Standards Guide which aims to build consensus among stakeholders from different backgrounds, provide a common language for understanding the lifecycle of health AI solutions and best practices for designing and deploying AI within healthcare workflows.
Policy and promoting innovation
A consistent framework for AI development sets industry standards, therefore fostering trustworthy innovation. To continue to encourage the development of AI, regulatory polices surrounding the technology must remain stable and reliable.
“One of the worst things that can happen to a technology innovator is investing in a first-of-its-kind technology that makes a meaningful difference in patient care, only to have the regulatory framework suddenly change midstream,” said Ingram. “A complete lack of regulations does not necessarily drive innovation; innovators instead want assurance that the rug won’t be pulled out from underneath their investments.”
In response to the need for coordinated efforts around AI in healthcare, organizations have come together to promote more responsible innovation.
“The success of innovation depends on the adoption, trust, impact, and use of technology, and CHAI emerged to fill this gap by building transparency, accountability, and fairness into the development and deployment of health AI from the start,” said Ghane.
Healthcare stakeholder coordination on the questions surrounding AI is crucial to ensure that the development and deployment of the technology is both consistent and trustworthy.
“Premier’s AI advocacy priorities center around responsible regulatory frameworks that include a call for transparency,” said Ingram. “As a company, we’re passionate about transparency of AI models being centered around the outputs of those models, as what you put into an AI model quickly transforms the model itself. We’re really interested in making sure model developers are accountable for outputs of their models.”
“The goal of the CHAI framework is to make responsibility operational. That means creating practical tools like testing and evaluation templates, model reporting formats, and risk categorization guidance that health systems, developers, and certifying bodies can use to make informed decisions and ensure safe, effective use,” said Ghane. “Ultimately, the framework aims to reduce friction in procurement and deployment while building public trust and advancing positive health access and outcomes for all.”
AI and improving patient outcomes
One of the many benefits of AI technology to the healthcare industry is its ability to optimize workflows and reduce administrative burdens, ultimately improving care delivery. Streamlined processes enable health systems to enhance patient outcomes and shift their focus toward value-based care.
“One neat thing that AI has the capability to do is go beyond the surface of a patient’s electronic health record (EHR) to look at unstructured data, clinical notes, images from radiology and more to glean things that might not necessarily be captured in the very structured fields of an electronic health record,” said Ingram.
AI is playing a significant role in the healthcare industry’s transition from fee-for-service to value-based care models by enhancing diagnostic accuracy, optimizing treatment plans and streamlining operations, all of which reduce health system expenses and improve care quality.
“Premier is passionate about the movement to value-based care, and the use of AI models for improving population health management pairs well with the movement to value-based care,” said Ingram. “When providers and payers are engaged in a value-based care arrangement, all parties have aligned incentives to drive the best outcomes for the patients. AI models that layer on top of the existing data sources really are the tool that makes a difference in catching health disparities in some of the most vulnerable populations.”
One of the most significant ethical challenges for healthcare stakeholders, however, is ensuring that AI tools don’t reinforce or exacerbate existing health disparities.
“It’s really important to ensure that both the public and private sector make adequate investments in workforce training so that the providers that are learning to use these new technologies in their clinical workflows don’t fall victim to automation bias or accidentally misusing a model for a purpose which it is not well suited,” said Ingram.
“Addressing AI ethics challenges requires careful attention to how models are trained, tested, and evaluated across diverse populations, and ensuring fairness is not an afterthought but a core design principle,” said Ghane. “This also comes up when we consider organizations that are better positioned to adopt and use AI solutions, and who have resources to effectively govern these solutions, versus those who could really benefit from these solutions, like smaller payers, community health centers, and rural hospitals, but that may be less resourced and struggle to develop the necessary infrastructure.”
By enabling access to data from diverse patient populations, instead of small data sets from a limited number of patients, physicians are able to expand their clinical capabilities, ensure that models are trained properly and support the ongoing development of AI models.
“It is important that any federal policy or regulatory structure incentivizes model developers to use robust, high-quality, real-world data to train their models,” said Ingram. “From a data perspective, when considering potential policy levers that can incentivize the integration of AI and a high-quality care, you must focus on model developers using data that’s generalizable to the national population.”
Flexible, but grounded
The effective regulation of healthcare AI requires collaboration across sectors. Consensus driven and layered approaches driven by state, local and national guidelines must create guardrails while maintaining regulatory structures that are flexible enough to continue to accommodate innovation, said Ghane.
“We’ve been using AI in various forms for many years to improve the healthcare system, and if we can find alignments and correct incentives, we can go farther in improving the patient experience, and the value of care that patients receive,” said Klein. “The push to improve patient outcomes while reducing unnecessary costs in the system is important, and at Premier, we want to see regulations that build on that desire in the health system, providers, and patients.”
“What we want to avoid is every state and local government establishing widely different rules, regulations, and definitions,” said Ghane. “This will hinder scalability for technology developers as well as those adopting these solutions.”
The regulation of AI should provide guidance without hindering innovation, and the effective management of AI technology involves regular reassessment of policy to ensure it is aligned with the evolving needs of the healthcare industry.
“The key is to create regulatory structures that are flexible enough to accommodate innovation, but grounded in clear standards for safety, transparency, usability, security, and fairness,” said Ghane. “Knowing that the resources we are developing include the voices and experiences of those using them is paramount.”
“When we see regulations that create actionable metrics, we introduce really useful data and information that produce useful, quality safeguards for AI technology,” said Klein. “Regulation shouldn’t exist for the sake of regulating; policies should instead have a clear purpose that improve the quality of care.”
The AI Lifecycle
Coalition for Health AI unveils 6-Stage Lifecycle for Responsible Health AI Development
Through a consensus driven process, the Coalition for Health AI (CHAI) has developed a 6-stage health AI lifecycle based on industry-standard AI development frameworks. The lifecycle encompasses processes from initial problem identification and solution planning through to large-scale deployment and monitoring of an AI solution and surrounding system (workflows, technical support components, personnel).
Stage 1: Define problem and plan
Healthcare is riddled with “solutions” in search of problems. Responsible innovation requires a clear understanding of the specific issue AI is intended to solve, which will drive the intended purpose of the tool. The intended purpose a) defines the scope of verification & validation activities and b) allows reasonable delineation between user responsibility and vendor responsibility. First, however, an upfront investment of time and effort is needed to map root causes and understand the specific needs of those experiencing the problem(s). Stage 1 focuses on this process, with developer teams conducting surveys, interviews or market research to understand healthcare system needs.
In Stage 1, implementer teams also identify a clear problem, its setting, and stakeholders involved (which together comprise a use case), thereby quantifying potential return on investment, return on health outcomes, and improvement of healthcare operations. Implementers use this information to decide whether to build an in-house AI solution, procure one from a third party, or partner with a third party to develop a solution.
Stage 2: Design the AI system
Stage 2 involves capturing details of the AI system of which the solution is the main component. This includes detailing its technical requirements, proposed system workflow, and deployment strategy. The design of the AI system is informed by the business requirements of the health system(s) and the needs of representative end users where the AI solution will be implemented.
Stage 3: Engineer the AI solution
The engineering stage aims to create an AI solution that can accurately predict or classify data and develop the interface for exposing AI solution output, as defined during the Design stage. This stage also ensures that AI solution deployment can be operationalized and that adequate planning is completed prior to deployment. In cases of externally developed AI solutions, the developer should provide expertise in collaboration with the implementer, who ensures that the AI solution meets its intended purpose via risk-benefit analysis before and after deployment.
Stage 4: Assess
This stage involves a series of assessments to determine whether to proceed with a pilot of the AI system in Stage 5. When existing AI technologies are acquired from an external organization, local validation and installation qualification need to be conducted first, prior to the assessment of the AI system as a whole. A change management plan should be in place to delineate who, between the developer and implementer, is responsible for performing these duties. This is followed by a prospective, silent evaluation and the establishment of a risk management plan, based on anticipated risks from Stages 2 through 4. These steps are followed by end user training and usefulness testing, along with a review to ensure compliance with applicable healthcare standards and regulations prior to piloting and deployment.
Stage 5: Pilot
The pilot stage is the first real-world use of the AI solution by the implementer team to inform large-scale deployment plans. Prior to the general deployment of an AI system, careful review and consideration must be made by the health system to decide whether or not to deploy an AI model into production. Based on this pilot stage, success criteria are reviewed to inform the decision on whether to deploy the AI system. Some common success criteria include the AI solution’s accuracy, reliability, interpretability, feasibility, user acceptance, cost, and alignment with the organization’s values and goals. This process is primarily undertaken by the implementer team.
Stage 6: Deploy and monitor
The deployment and monitoring stage is the process of making the AI solution and system broadly available to the healthcare system or relevant specialty. Once deployed by the implementer team, the AI solution is often handed over to a model operations team (when available) to provide ongoing monitoring, retraining, and governance of models to ensure peak performance and that decisions
are transparent.
Source: https://chai.org/workgroup/responsible-ai/responsible-ai-guide-raig-and-raig-executive-summary