Societal Implications › Governance Frameworks

Beyond government regulation, AI governance encompasses a broader ecosystem of standards, self-regulatory mechanisms, technical tools, and institutional practices. Understanding this ecosystem is important because regulation alone is insufficient, too slow to keep pace with AI development, and limited in technical specificity.

The governance ecosystem is best understood as a series of layers, each addressing different aspects of the problem. Technical documentation practices create transparency. Auditing frameworks provide external verification. Sector regulations address high-stakes applications. National legislation sets outer limits. International coordination attempts to ensure global coherence. No single layer is sufficient; effective governance requires all of them to function reasonably well simultaneously.

Several technical approaches have been developed to make AI systems more governable.

Model cards are standardized documentation for AI models that specify intended use cases, training data characteristics, performance across demographic groups, and known limitations. Proposed by Mitchell et al. (2019) at Google, model cards have been adopted by major AI providers and are increasingly required by deployment platforms. They serve as a form of technical transparency that enables informed deployment decisions. A model card cannot substitute for regulatory requirements, but it provides the information that regulators, deployers, and users need to make informed decisions about whether and how to use a system.

Datasheets for datasets, an analogous concept proposed by Gebru et al. (2018), document the characteristics, collection methodology, and potential biases of training datasets. The argument is that many downstream harms from AI systems can be traced to undocumented characteristics of training data, and systematic documentation is a prerequisite for addressing them. If the characteristics of training data are not documented, practitioners downstream cannot know what population the data represents, what biases it may contain, or what use cases it is inappropriate for.

Auditing frameworks provide structured methodologies for evaluating AI systems for bias, accuracy, and safety properties. Academic researchers, standards bodies, and civil society organizations have developed audit methodologies for various domains. Third-party auditing of high-stakes AI systems has been proposed as a regulatory requirement in several jurisdictions, analogous to financial auditing requirements for public companies. The analogy is instructive: just as financial auditing does not guarantee that companies are well-managed, AI auditing does not guarantee that systems are safe, but it does create an external accountability mechanism and a disincentive to obvious failures.

Red teaming refers to systematic adversarial testing of AI systems to identify failure modes before deployment. As discussed in the alignment and safety sections of this handbook, red teaming has become a standard component of responsible deployment for frontier models. Institutionalizing red teaming practices and sharing findings across the industry could improve the overall safety of AI deployments. The challenge is that red teaming findings create competitive intelligence about model weaknesses, which creates incentives to keep them proprietary rather than share them with competitors. Governance structures that create safe channels for sharing safety-relevant information without creating competitive disadvantage could address this incentive problem.

Evaluation benchmarks deserve mention as a governance tool because they create shared standards for assessing AI capabilities and limitations. When regulators, deployers, and the public can refer to standardized benchmark results, they can make more informed decisions about deployment. The limitation of benchmarks, as discussed throughout this handbook, is that they measure performance on specific test distributions that may not reflect real-world deployment conditions. Benchmark gaming, where models are optimized for benchmark performance rather than general capability, is a real concern. Nevertheless, well-designed benchmarks that are updated as capabilities evolve are a useful component of the governance toolkit.

Major AI companies have developed voluntary governance commitments, including safety policies, responsible AI principles documents, and voluntary commitments made to governments. In July 2023, major AI companies including OpenAI, Google, Microsoft, Anthropic, and others made voluntary commitments to the Biden White House on AI safety, including commitments to share safety information, implement red teaming, develop technical mechanisms to indicate AI-generated content, and prioritize research on AI risks.

The question of whether industry self-regulation is adequate, and under what conditions it should be complemented by or replaced by binding regulation, is one of the central contested questions in AI governance. The answer likely varies by application domain and the severity of potential harms. For high-stakes applications in healthcare, criminal justice, and critical infrastructure, the consensus view among governance researchers increasingly favors binding external requirements rather than voluntary commitments.

A useful framework for evaluating voluntary commitments is to ask what would change if the commitment were not made. If a company commits to doing something it would do anyway because it is in its commercial interest, the commitment adds little. If a company commits to doing something it would not otherwise do, the commitment represents an enforceable governance mechanism, though one whose durability depends on the continuing alignment of commercial incentives. The most valuable voluntary commitments are those that credibly bind the committing organization to practices that might become commercially inconvenient, and that include verification mechanisms so the commitment is not merely aspirational.

Effective AI governance requires more than government regulation and industry self-regulation. It requires mechanisms for civil society participation, democratic accountability, and informed public discourse.

Civil society organizations play several roles in the AI governance ecosystem. Technical advocacy organizations like the AI Now Institute, the Algorithmic Justice League, and the Center for Democracy and Technology conduct research on AI harms, develop policy proposals, and advocate for affected communities. Legal organizations bring litigation to challenge harmful AI deployments. Community organizations representing affected populations bring lived experience to governance deliberations.

The capacity constraints of civil society deserve acknowledgment. The organizations best positioned to conduct sophisticated technical analysis of AI systems often lack resources to engage consistently in governance processes. The organizations with the deepest connections to affected communities often lack technical capacity to evaluate AI systems rigorously. Building civil society capacity to participate effectively in AI governance is itself a governance challenge that philanthropy and public funding have only partially addressed.

Democratic accountability for AI requires that citizens have access to meaningful information about how AI systems affect their lives, have recourse when harmed, and have some mechanism for influencing AI deployment decisions that affect public welfare. Several mechanisms have been proposed:

• Algorithmic impact assessments: Requirements for public institutions to assess and disclose the potential impacts of AI deployments before implementation, analogous to environmental impact assessments. The analogy is instructive because environmental impact assessments were initially resisted by industry as burdensome and have become a normal part of infrastructure planning.
• Procurement requirements: Requirements for public sector AI procurement to meet specific standards for safety, fairness, and transparency. Governments are large AI purchasers and can use procurement to drive industry standards.
• Appeal rights: Requirements that consequential AI-based decisions be subject to human review when contested. The right to human review of automated decisions is increasingly recognized in regulatory frameworks including the EU AI Act and the EU GDPR.
• Public registries: Databases of AI systems used in public-sector contexts, enabling public scrutiny. Several jurisdictions have begun requiring registration of high-risk AI systems.

The tradeoffs revealed here explain why effective governance tends to combine multiple approaches. Voluntary commitments and technical documentation can be implemented quickly and are technically specific, but lack enforcement power and democratic accountability. National legislation provides enforcement and accountability but takes years to develop and may become outdated as technology evolves. Sector-specific regulation can develop faster than national legislation and applies domain expertise, but creates gaps between sectors. Impact assessments score well on democratic accountability but require significant institutional capacity to implement well. The strongest governance systems layer multiple mechanisms, using faster technical approaches for immediate action while developing more durable regulatory frameworks over time.

The most difficult governance challenges arise at the international level, where national regulatory authority meets global deployment reality.

Jurisdictional arbitrage is a real concern. If AI systems that do not meet EU standards can be developed in less-regulated jurisdictions and offered to EU users through the internet, EU regulation achieves less. This has motivated discussions about extraterritorial application of AI rules (the EU AI Act applies to any AI system with effects in the EU regardless of where it was developed) and about mutual recognition agreements between jurisdictions. The challenge is that extraterritorial claims create diplomatic friction and are difficult to enforce when the developing organization has no presence in the regulating jurisdiction.

Frontier model governance raises distinct challenges. The most powerful AI systems are developed by a small number of organizations in a few countries, primarily the US and China. The governance of these systems, which could have large effects on society, cannot be left entirely to national regulation of the developing country. International mechanisms for ensuring that frontier AI systems meet minimum safety standards, and for sharing safety-relevant information across jurisdictions, are under active development.

The Bletchley Declaration on AI safety (2023) established international consensus that frontier AI poses serious safety risks that require international coordination, though it stopped well short of binding commitments. Subsequent discussions at the Seoul AI Safety Summit and Paris AI Action Summit have advanced the conversation but have not yet produced durable international governance mechanisms. The gap between the urgency expressed in these forums and the binding commitments that result from them remains wide.

AI and geopolitics adds a layer of complexity. AI capability is increasingly viewed as a national security asset, creating tensions between international information sharing for safety purposes and national interests in maintaining strategic advantages. The US has imposed export controls on advanced AI chips to limit China's AI development capacity. These measures reflect a view of AI capability as geopolitical power, which complicates cooperation frameworks that assume shared interests in safety. It is possible to simultaneously believe that AI poses safety risks that require international cooperation and that AI capability represents a strategic resource worth protecting from rivals; the difficulty is that these beliefs point in opposite directions for governance design.

One model for managing this tension comes from nuclear governance, where countries with strategic nuclear interests have nonetheless participated in international safety frameworks. The analogy is imperfect in many ways, but the principle that strategic competition and safety cooperation can coexist provides some basis for optimism about international AI governance, even in a geopolitically competitive environment.