Center for AI Risk Management & Alignment

Coming to high-level consensus on red lines as regards maximum allowable risk from each AI system, per best practice for prospective AI risk assessment.

• Safe, secure and trustworthy AI
• Social, economic, ethical, cultural, linguistic and technical implications of AI
• Transparency, accountability, and human oversight
• Interoperability of governance approaches

These four priorities reflect the convergent pressures that make AI governance both urgent and structurally difficult. Safe, secure and trustworthy AI is foundational: without robust progress on safety assurance mechanisms that keep pace with rapidly advancing capabilities, all other governance objectives rest on unstable ground. Transparency, accountability, and human oversight is its necessary complement, since safety claims that cannot be independently verified become a vector for safety-washing, eroding public trust and regulatory credibility alike. Interoperability of governance approaches addresses what may be the single greatest structural vulnerability in the current landscape: fragmented national and regional frameworks create regulatory arbitrage opportunities that incentivize a race to the bottom, while simultaneously leaving gaps through which systemic and cross-border risks propagate unchecked. Finally, the social, economic, ethical, cultural, linguistic and technical implications of AI must be treated as inseparable from safety considerations rather than as a separate track, because AI systems that are nominally "safe" in a narrow technical sense can still concentrate power, deepen inequality, and erode the societal resilience on which effective crisis response depends. Taken together, these four areas form an interlocking set: meaningful progress on any one of them requires, and reinforces, progress on the others, and neglecting any one creates failure modes that undermine the rest.

Catastrophic risk from advanced AI and the pace of capability development. The current thematic framework implicitly assumes AI systems that remain tools under meaningful human direction. Yet the most consequential near-term development in AI is the rapid emergence of increasingly autonomous, general-purpose AI agents capable of planning, acting across domains, and resisting correction. The governance challenges posed by such systems are qualitatively different from those posed by narrow AI, and they cut across every listed theme simultaneously. Three specific gaps deserve attention. First, the speed of frontier AI capability growth is structurally outpacing governance capacity at every level. Governance frameworks designed around consultation timelines of years face AI capability doublings measured in months. Without mechanisms for adaptive, rapid-response governance, even well-designed frameworks risk being obsolete upon adoption. Second, concentration of transformative AI capability in a small number of private actors and states represents a cross-cutting structural risk. This concentration creates single points of failure, power asymmetries that undermine multilateral governance legitimacy, and incentive structures that reward speed over safety. It intersects with but is not reducible to existing themes around equity or the digital divide. Third, compounding and systemic risk across domains is inadequately addressed by a thematic structure that treats AI's implications sector by sector. AI systems increasingly mediate critical infrastructure, financial systems, information ecosystems, and scientific research simultaneously. A disruption, failure, or loss of control in one domain can cascade unpredictably across others. Governance must account for these interaction effects, not merely manage individual domains in isolation. Finally, the international community should explicitly address emergency preparedness and societal resilience in the context of acute AI-driven crises, whether from accident, misuse, or loss of control. Civil defense, biosecurity, and critical infrastructure hardening against AI-enabled threats deserve standing attention within the dialogue rather than treatment as peripheral concerns.

As a civil society organization focused on catastrophic AI risk, we operate at the intersection of technical safety research, public policy, and emergency preparedness. Governance gaps in the thematic areas we identified affect our sector in concrete and compounding ways. The absence of enforceable international safety standards means that frontier AI developers self-certify the safety of increasingly powerful systems. For organizations working to assess and mitigate AI risks, this creates a fundamental information asymmetry: we lack the access, audit authority, and standardized evaluation frameworks necessary to independently verify safety claims. Safety-washing, where voluntary commitments substitute for binding obligations, has become a persistent challenge that erodes both public trust and the credibility of genuine safety efforts. Fragmented governance across jurisdictions directly undermines our work. Most AI risks do not respect borders, yet organizations developing dangerous AI capabilities can select the most permissive regulatory environment. This regulatory arbitrage accelerates deployment timelines while narrowing the window for meaningful safety evaluation. Compounding this, there is a critical lack of political will, rooted largely in insufficient technical literacy among policymakers, to establish and enforce verification regimes governing frontier AI compute. The physical infrastructure required to train the most powerful AI systems is concentrated, observable, and verifiable, yet the international community has not seriously entertained the monitoring frameworks that this concentration makes feasible. This represents a squandered governance advantage that will not persist as compute becomes more distributed and algorithmic efficiencies reduce hurdles to calamity. The pace of AI potency advancement represents the most acute challenge. Governance deliberation operates on timescales of years; AI capabilities advance on timescales of months. Our sector increasingly faces the prospect that systems posing novel catastrophic risks will be deployed before adequate governance frameworks exist. The principal opportunity is that international awareness of these risks is growing, creating genuine political space for ambitious action. This window is time-limited, however: once sufficiently advanced systems are deeply embedded in economic and military infrastructure, the feasibility of meaningful constraint diminishes sharply.

The AI Dialogue occupies a structurally unique position in the international governance landscape: it is one of the few venues where member states, civil society, the private sector, and technical communities engage collectively on AI governance under the legitimating authority of the General Assembly. That position confers both a distinctive opportunity and a distinctive responsibility. The Dialogue's most valuable potential contribution is norm convergence: not imposing a single regulatory model, but establishing shared baseline expectations around transparency, safety evaluation, and accountability that can anchor diverse national frameworks and reduce the most dangerous forms of regulatory arbitrage. Achieving this requires the Dialogue to move beyond declaratory consensus toward operationally specific commitments. Critically, the Dialogue should prioritize building the technical and institutional infrastructure for verification. Existing multilateral arms control and nonproliferation regimes demonstrate that international verification is achievable when there is political will and appropriate institutional design. The compute infrastructure underlying frontier AI development is, at present, sufficiently concentrated and physically locatable to make meaningful monitoring feasible. The Dialogue can build political will for such mechanisms by commissioning technically credible feasibility assessments and making the case to member states that verification is achievable rather than aspirational. The Dialogue should also function as an early warning and rapid response forum, not merely a periodic convening. AI capability advances too quickly for governance that meets on annual or biennial cycles to remain relevant. Establishing a standing capacity to identify emerging risks, surface technical developments with governance implications, and convene rapid expert consultation would significantly strengthen the international community's ability to respond before risks become entrenched. Finally, the Dialogue can serve a crucial capacity-building function, ensuring that smaller states and civil society organizations have the technical literacy and participatory access necessary to engage substantively rather than deferring to the framing provided by powerful actors with vested interests in particular governance outcomes.

Several existing initiatives provide foundations the AI Dialogue should actively build upon rather than duplicate, leveraging its unique General Assembly mandate to connect and strengthen them. The Global Digital Compact and the Pact for the Future, from which the Dialogue itself originates, established commitments that now require operational follow-through. The companion UN Independent International Scientific Panel on AI, designed as an evidence and early-warning body, should be tightly integrated with the Dialogue's deliberations so that governance discussions are continuously informed by independent scientific assessment rather than proceeding on parallel tracks. The AI Safety Summit process (Bletchley, Seoul, Paris, and the forthcoming India AI Action Summit) has catalyzed the creation of AI Safety Institutes across multiple countries. The Dialogue can provide a formally multilateral and more inclusive complement to this process, anchoring its outputs into a framework where developing nations have equal standing rather than observer status. The Pro-Human AI Declaration (March 2026), endorsed by a broad coalition including Turing Award laureate Yoshua Bengio, Nobel laureate Daron Acemoglu, and organizations spanning labor, civil society, and technology, articulates principles on human control, accountability, and the prevention of power concentration that align directly with the Dialogue's thematic priorities. The Dialogue should recognize and engage with this growing pro-human movement as a source of normative legitimacy and civil society energy. The OECD AI Principles, UNESCO's Recommendation on the Ethics of AI, and the Council of Europe AI Convention provide complementary normative and legal frameworks. The ITU's AI for Good platform, alongside which the Dialogue will convene in Geneva, offers a natural operational partner. The Dialogue's distinctive added value lies in its universality: it is the only venue where every country has a formal seat. That legitimacy should be deployed to broker convergence among these fragmented initiatives, identify the gaps none of them address (particularly around verification and enforcement), and provide political weight to governance proposals that have stalled elsewhere. As officials have described it, the Dialogue should function as a "dialogue of dialogues," connecting existing efforts into a coherent whole rather than centralizing governance under a single framework.

The Dialogue's value will be determined not only by the breadth of its participation but by whether its structure enables that participation to produce actionable outcomes rather than just declaratory statements. Governments should contribute binding national commitments and concrete governance proposals rather than restatements of principle. The Dialogue should require participating states to submit national AI risk assessments and governance implementation reports, creating a peer review mechanism analogous to the Universal Periodic Review in human rights. This introduces some accountability without requiring treaty-level enforcement. Civil society and academia should serve as independent evaluators and early warning voices, not merely as consultation participants whose input is acknowledged but structurally sidelined. The Dialogue should guarantee civil society allocated speaking time, formal written response rights to government and industry proposals, and seats on any technical working groups. Organizations focused on catastrophic risk, labor, human rights, and Global South digital equity each bring perspectives that must be structurally embedded rather than tokenized. The private sector, particularly frontier AI developers, should be expected to contribute transparency disclosures on capabilities, safety testing methodologies, and compute usage as a condition of meaningful participation. Voluntary pledges without verification have proven insufficient; the Dialogue should establish expectations that private sector engagement entails substantive information sharing, not merely presence. The Independent International Scientific Panel on AI should present its assessments in dedicated sessions with structured government response periods, ensuring its findings are formally received and addressed rather than merely noted. The Dialogue should adopt an intersessional working group model rather than relying solely on annual plenary sessions. AI governance cannot advance meaningfully through two-day annual meetings alone. Standing working groups on specific thematic areas, empowered to produce draft recommendations between sessions, would maintain momentum and enable the kind of technical depth that plenary formats preclude. The Dialogue should also establish a public submission and response mechanism that operates year-round, ensuring continuous stakeholder input rather than episodic consultation.

The most consequential underrepresentation in global AI governance is not a matter of which groups lack seats at the table, but of which groups lack the structural capacity to influence outcomes once seated. Communities experiencing AI's harms firsthand are systematically underrepresented: gig workers subject to algorithmic management, populations in conflict zones where autonomous systems are deployed, communities subjected to biometric surveillance, and individuals whose livelihoods are being displaced by automation. These groups possess direct experiential knowledge that no technical expert or government delegate can substitute for, yet they rarely have the resources, institutional affiliations, or travel funding to participate in international fora. Critical professional communities whose expertise is essential to governing AI's societal impact are largely absent from these discussions. Disaster preparedness and emergency management professionals understand systemic failure and cascading risk in ways directly applicable to AI-driven crises. Sociologists can assess the structural transformations AI is driving in ways technologists cannot. Children's safety analysts bring urgency to questions about developmental harm from AI-mediated environments. Risk managers across insurance, finance, and critical infrastructure possess mature frameworks for quantifying and mitigating exactly the kinds of compounding, tail-risk scenarios that advanced AI introduces. These disciplines are not peripheral; they are foundational to credible governance. Workers and organized labor, despite being among the populations most immediately affected by AI deployment, remain marginal participants in governance processes dominated by technology firms and government ministries. The recent mobilization around the Pro-Human AI Declaration demonstrates that labor voices are organizing with real momentum, but institutional channels for their input into multilateral AI governance remain thin. The Dialogue should treat labor not as a stakeholder to be consulted but as a co-designer of governance frameworks addressing automation, workplace surveillance, and economic disruption. Independent AI safety researchers and whistleblowers possess critical knowledge about frontier system risks but face structural barriers to candid participation, including nondisclosure agreements, employment dependencies on the very companies whose systems require scrutiny, and credible risks of professional retaliation. Without protected channels for their testimony, the Dialogue risks hearing only from those whose candor is constrained by commercial interest. Inclusion requires structural remedies, not invitations. The Dialogue should establish a dedicated participation fund, mandate multidisciplinary co-design of session agendas, create protected channels for confidential expert testimony, and ensure interpretation extends beyond the six UN languages to encompass the linguistic communities most affected by AI's limitations.

Traditional plenary formats produce traditional outcomes: broad consensus statements that lack operational specificity. If the AI Dialogue is to generate governance that keeps pace with AI development, its engagement formats must be as adaptive as the technology they address. Structured adversarial exercises. The Dialogue should convene scenario-based stress tests in which mixed teams of policymakers, technologists, civil society representatives, and risk professionals confront simulated AI governance crises: a frontier model deployment that triggers cross-border harm, a loss-of-control incident at a major lab, a cascading infrastructure failure mediated by autonomous systems. These exercises expose governance gaps in ways that abstract discussion cannot, build shared situational awareness across stakeholder groups, and generate concrete recommendations grounded in operational realism rather than theoretical principle. Red-teaming of governance proposals. Before any framework or recommendation is finalized, it should undergo structured adversarial review by teams explicitly tasked with identifying failure modes, circumvention strategies, and unintended consequences. This should include representatives from regulated entities, affected communities, and independent risk analysts. Governance proposals that cannot survive structured critique should not advance. Rapid-response technical briefings. Between formal sessions, the Dialogue should host short-turnaround briefings when significant AI capability advances or incidents occur, convened within weeks rather than months. These should pair technical explanation with immediate governance implication analysis, preventing the accumulation of unaddressed developments between annual meetings. Fishbowl dialogues pairing affected communities with developers. Rather than segregating stakeholder input into separate consultation tracks, the Dialogue should create structured encounters where gig workers, content moderators, safety researchers, and children's advocates engage directly with frontier AI developers in formats that require substantive response rather than acknowledgment. Living documents with public annotation. Draft recommendations should be published on open platforms allowing structured public commentary, with formal obligations to address substantive input, transforming the Dialogue from a periodic event into a continuous governance process.

Several concrete approaches demonstrate that effective AI governance is achievable and offer models the Dialogue should promote. Tiered assessment and runtime governance for autonomous AI agents. An emerging international consensus identifies foundational principles for managing agentic AI risk, including least privilege, traceable identity, auditability, validated deployment, adversarial resilience, runtime assurance, interruptibility, legibility, and human-in-the-loop oversight. These are not aspirational: convergent implementation across frontier developers and multiple jurisdictions shows they are already operationalized in practice and ready for broader adoption. Architectural security controls as primary defenses. Deterministic, model-external controls must serve as primary safeguards for advanced AI systems, because AI-based defenses where one model supervises another break down predictably when the supervised model exceeds the monitor's capabilities. Non-reasoning hardened layers that cannot be deceived provide a governance template applicable across jurisdictions and development contexts. Probabilistic risk assessment adapted for AI. Techniques drawn from high-reliability industries such as nuclear energy, aviation, and chemical safety can be adapted for AI systems through aspect-oriented hazard analysis, risk pathway modeling, and structured uncertainty management, enabling systematic, documented risk assessment rather than ad hoc evaluation. Societal offense-defense dynamics as an evaluation lens. Rather than assessing AI systems solely through narrow technical safety benchmarks, governance should evaluate whether a given system is likely to strengthen or degrade societal resilience given its capabilities, accessibility, and deployment context. The question is not "what can this model do?" but "what is this model likely to cause?" National preparedness and civil defense frameworks. Existing emergency response systems are not designed for the speed, scale, and technical complexity of AI-driven crises. Concrete governance advances include mandating incident reporting, strengthening institutional preparedness, and developing AI-specific emergency protocols.