Duke University School of Medicine, Duke Health AI Evaluation & Governance Program

A successful Dialogue should move the conversation from principles to practice. There is no shortage of frameworks describing what responsible AI should look like. What is still missing, especially in healthcare, is clarity on how AI governance should be set up and how it operates once AI is in use. At Duke Health, and through our work with other health systems, we have focused on what it takes to operationalize governance in real clinical environments. AI tools are already embedded in workflows and influencing patient care, which means governance must function as part of the system itself. In practice, this includes a centralized intake and registry of AI solutions, risk-based evaluation before deployment, and multidisciplinary review involving clinical, technical, regulatory, and operational expertise. Just as important is what happens after deployment. Tools are continuously monitored for performance, bias, and workflow impact, with issues identified and routed through existing patient safety and quality systems. We have learned that governance is most effective when it builds on existing infrastructure rather than creating parallel processes, and when it balances rigor with usability so that it supports, rather than slows, adoption. What would make this Dialogue meaningful is a stronger focus on how this work happens in practice. It should create space to share lessons learned, including what has been difficult or unexpected, and help organizations move from high-level guidance to approaches they can realistically implement. The most valuable outcome would be a clearer understanding of how to set up and sustain AI governance in real-world settings, informed by shared lessons from implementation and a better understanding of how health systems at different levels of maturity can adopt AI responsibly.

• Safe, secure and trustworthy AI
• Social, economic, ethical, cultural, linguistic and technical implications of AI
• Interoperability of governance approaches
• Transparency, accountability, and human oversight

Our selections reflect where governance moves from theory into practice. In a health system, the question is not whether AI is trustworthy in theory, but whether it remains trustworthy once it is in use. As adoption accelerates, many health systems are still not equipped to consistently identify when AI is involved in a patient safety event or to track its impact over time. At Duke Health, we have seen that performance can change and tools can behave differently across patient populations or clinical settings. This is why safety and trust cannot be treated as a one-time checkpoint. They require ongoing attention. Transparency and accountability become practical concerns in this environment. Clinicians need to understand what a tool is designed to do, how to interpret its outputs, and when to rely on it. Organizations also need clarity on who owns each system, who is responsible for testing and monitoring it, and how concerns are escalated and addressed. This is especially important for third-party tools, where responsibility can be less clearly defined. Interoperability is a growing priority. Many tools are developed outside the health system and deployed across multiple organizations. Without some consistency in governance approaches, each institution is left to interpret risk and oversight independently. Finally, the broader implications of AI show up in daily use. At Duke Health, we have observed bias across patient populations, automation bias in decision-making, and shifts in patient engagement. These issues are often not captured through traditional evaluation and can be difficult to detect without dedicated monitoring. Many risks emerge over time rather than as single events, which has shaped our approach to ongoing surveillance and risk management.

At the organizational level, AI governance should be understood as a people, process, and technology framework that enables the safe, effective, and ethical use of AI. Governance serves as the structure for oversight, but it must be supported by complementary functions such as risk management, monitoring, safety reporting and surveillance, independent review, and workforce training. Together, these elements determine whether governance is meaningful in practice. One emerging issue is how to manage AI that changes over time. Many approaches still assume models are stable once deployed. In practice, models may degrade, be updated, or behave differently as clinical patterns shift. At Duke Health, this has required a shift from one-time approval to ongoing stewardship, where performance is continuously assessed and verified. As more advanced systems, including agentic AI, enter clinical settings, the need for human oversight, automated evaluation, and clear patient disclosure becomes even more important. Another challenge is the growing distance between developers and users. Tools are often built externally and introduced into environments with very different workflows and constraints. Governance must bridge this gap by evaluating not only the model, but how it performs in real-world use. Visibility is also a persistent issue. Many organizations lack a complete inventory of AI tools, making it difficult to monitor performance or manage risk consistently. The impact of these gaps is significant. Without effective governance, risks may go undetected, trust may erode, and adoption may slow. When governance is in place, it creates the conditions for safer, more consistent, and more scalable use of AI.

In healthcare, governance gaps show up in both subtle and visible ways. Depending on the size and resources of a health system, the processes and expertise needed to evaluate AI may not be in place, creating uneven capacity to deploy AI responsibly and widening disparities in access to trustworthy solutions. AI also introduces a structural challenge. Model performance can vary across patient populations and over time, yet health systems remain accountable for outcomes. In our experience within a large academic U.S. health system, AI tools are increasingly embedded in clinical workflows, but safety oversight has not fully kept pace. At Duke Health, we saw early on that traditional quality structures were not designed to track how algorithms behave over time or how they influence clinician decision-making. In the absence of tailored governance, two patterns emerge: AI is used without sufficient visibility into performance, or adoption slows because clinicians lack confidence in oversight. Both create risk, either through blind spots or missed opportunities. A central challenge is designing governance that can keep pace with a fast-moving environment. It must balance rigor with usability. If too heavy, it slows adoption; if too light, it misses risks. Scale adds complexity, as processes that work for a few models become strained as adoption grows. The rapid shift from large language models to more autonomous systems illustrates how quickly governance needs evolve. At the same time, there are clear opportunities. Governance provides structure, improves visibility into AI use, and builds trust among clinicians and operational teams. It also creates a foundation for learning across institutions, where shared approaches can reduce duplication and accelerate progress. As health systems move from pilots to broader deployment, governance becomes essential. Done well, it enables AI to move from isolated innovation to safe, consistent scale.

The Dialogue can connect groups working on different parts of the same problem across regions and disciplines, including data science, informatics, medicine, ethics, and policy, spanning industry, academia, and government. Today, policy discussions, technical development, and implementation often move in parallel and within regional silos. At Duke Health, many of the practical questions we face are not fully addressed in existing frameworks, in part because those frameworks are not grounded in day-to-day operational experience. The Dialogue can bridge these gaps by bringing perspectives together across geographies. It can enable implementers to share how governance works in practice, while helping policymakers understand where guidance is clear and where it leaves gaps. This exchange can better align expectations with what is feasible, especially as regions adopt AI at different speeds. It can also strengthen cooperation by identifying where approaches are already converging. While health systems in the United States and Europe operate under different regulatory environments, similar patterns are emerging around lifecycle oversight, monitoring, and accountability. Highlighting these common elements can support more consistent and adaptable governance approaches. At the same time, the Dialogue should recognize differences in infrastructure and readiness. Some regions are still building the capacity to support AI governance. There is an opportunity for more established systems to share practical approaches and lessons learned, helping others adapt governance to local contexts without starting from scratch. Finally, fostering a more open culture of international exchange will be key. Governance improves when institutions learn from each other's experiences, including challenges, and the Dialogue can create a space where that exchange is expected and supported.

There are already strong efforts shaping AI governance at the policy level, including work from international bodies and U.S. regulators such as the National Institute of Standards and Technology and the U.S. Food and Drug Administration, alongside multi-stakeholder initiatives like the Coalition for Health AI and the Trustworthy and Responsible AI Network. In parallel, health systems such as Duke are building internal governance processes to manage AI in practice. At Duke Health, much of our work has focused on translating broad principles into operational steps, including how tools are submitted for review, how risk is assessed, and how performance is monitored once deployed. Through collaborations with other health systems and participation in national initiatives, we see similar efforts emerging, often in parallel and with limited coordination across regions. The AI Dialogue can add value by connecting these efforts more deliberately. For example, it could help align work from U.S.-based initiatives with regulatory approaches in other regions, including agencies such as the European Medicines Agency, to identify shared expectations for evaluation and post-deployment monitoring. There is also an opportunity to move toward a more common foundation for governance, similar to how global efforts have aligned around environmental or public health priorities. Even partial alignment on elements such as AI system registries, lifecycle oversight, and monitoring practices would make it easier for institutions to adopt and scale governance, particularly when tools are deployed across multiple settings. Through the Duke Health Evaluation and Governance program, we have identified recurring themes in AI oversight, particularly in governance design and safety monitoring, which have informed a set of practical policy recommendations. By building on existing initiatives and connecting them across regions, the Dialogue can help shift from fragmented efforts to a more coordinated approach to implementation.

The Dialogue will be most effective if it reflects how decisions about AI are actually made, both at a system level and within specific sectors. In practice, governance decisions are not made by a single group. At Duke Health, AI governance involves representative end users across roles; in healthcare, that includes physicians, nurses, pharmacists, and other care providers, alongside data scientists, operational leaders, and informatics teams. Each brings a different view of how AI affects care delivery. At the same time, there are cross-cutting governance themes that apply across sectors, including safety, accountability, transparency, and ongoing monitoring. These shared challenges create an opportunity for alignment. However, how these principles are applied can vary by industry. In healthcare, governance is closely tied to patient safety, clinical workflows, and regulatory oversight, while other sectors may prioritize different risks and outcomes. The Dialogue should reflect both dimensions. It should identify common governance foundations while also allowing for sector-specific discussions that account for differences in context and maturity. Structurally, sessions could focus on shared challenges such as monitoring models over time or defining accountability, with perspectives from multiple sectors, alongside sector-focused discussions grounded in real use. The format should emphasize discussion over presentation. Smaller groups, moderated exchanges, and approaches such as modified Delphi can support more practical insight and consensus. Outputs should remain open for public comment. Ahead of the Dialogue, surveying a broader community could help ground discussions and ensure a wider range of perspectives are represented.

Many discussions on AI governance are shaped by those who design or regulate systems, but less by those who use and manage them day to day. At Duke Health, frontline clinicians have been essential in identifying how AI tools actually perform in practice. This includes typical end users who experience how these tools fit into workflows. They see where AI adds value, but also where it introduces friction, uncertainty, or unintended consequences. Their perspective often differs from what is assumed during development. Operational teams who implement and support these tools are also critical. They understand the realities of integration, training, and ongoing monitoring, and often surface issues that are not visible at the design stage. Across different sectors, representatives of those impacted by AI are another important voice to include. In healthcare, AI directly affects care decisions, yet patient perspectives are not consistently included in governance discussions, particularly when it comes to trust, communication, and access. There is also a need to more intentionally include perspectives that are currently underrepresented. Women remain underrepresented in many AI leadership and technical spaces, despite being a significant part of the healthcare workforce and patient population. Similarly, communities that are more likely to be affected by the digital divide, including those in lower-resourced settings, risk being left out of both design and governance conversations. Ensuring these voices are present requires more than open invitations. It means actively designing participation to reflect lived experience and diverse perspectives. This will lead to governance that is more representative, more practical, and more equitable.

Formats that focus on real situations tend to produce more meaningful discussion. Discussions can be grounded findings from the literature and presentations from experts who have implemented AI governance in the real-world across different industries (healthcare, aviation, agriculture, banking, etc.), showcasing components of AI governance. Common and industry specific themes and priorities can be discussed in breakout sessions, where small groups can work through how they would approach governance in different contexts. This encourages active participation and surfaces different perspectives. There is also value in making room for honest reflection. Sessions focused on lessons learned, including what did not work as expected, can provide insight that is often missing from formal presentations. At Duke Health, much of our learning has come from working through specific cases. For example, introducing a new model into a clinical workflow often raises questions that are not obvious in advance. Discussing those scenarios in detail is often more useful than reviewing general principles. The Dialogue could reflect this by incorporating case-based sessions where participants walk through actual examples. This could include what decisions were made, what challenges arose, and how they were addressed. Volunteers can present different use cases for discussion and polling on priority use cases base on industry/sector would be beneficial. These types of formats can make the Dialogue more interactive and more grounded in practice.

The Duke Health AI Evaluation and Governance Program's ABCDS Oversight initiative provides a practical example of how governance can be implemented within a health system (program website: https://healthaigovernance.duke.edu). At Duke Health, AI tools are brought through a centralized intake process where their intended use and potential impact are reviewed, ensuring consistent evaluation and visibility across the organization (JAMIA Paper: Introducing Framework for Clinical Algorithm Oversight https://academic.oup.com/jamia/article-abstract/29/9/1631/6596175). This approach is formalized through an enterprise AI oversight policy and standard operating procedures, and we would welcome the opportunity to share these resources with the global community. Once deployed, tools are continuously monitored for performance, bias, and workflow impact, with oversight integrated into existing patient safety and quality systems. This allows issues to be identified and addressed within familiar structures. This approach is complemented by broader efforts to define and share practical governance models. Duke Health has contributed to multi-stakeholder initiatives such as Coalition for Health AI (CHAI), helping develop responsible AI guidance for healthcare (https://www.chai.org/workgroup/responsible-ai/responsible-ai-guide-raig-and-raig-executive-summary) that brings together perspectives from industry, academia, and government. We have also contributed to AI governance and safety white papers that have informed emerging policy recommendations (White Paper: AI Governance in Health Systems: Aligning Innovation, Accountability, and Trust: https://healthpolicy.duke.edu/publications/ai-governance-health-systems-aligning-innovation-accountability-and-trust and White Paper: AI Safety in Health Systems: Building Infrastructure and Strengthening Risk Management Practices: https://healthpolicy.duke.edu/publications/ai-safety-health-systems-building-infrastructure-and-strengthening-risk-management). We have also developed evaluation frameworks and implementation guidance (https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocad221/7455696). In parallel, tools such as health AI maturity models and implementation playbooks are helping organizations assess readiness and build governance capacity. A key lesson is that governance is not a single decision point but an ongoing process requiring continuous monitoring, clear accountability, and integration into real-world systems.