Warnings of the perils of artificial intelligence are spreading like, well, autonomous AI agents going rogue. Likewise are opinions on how to rein it in – or not. A leading voice from one highly regulated industry suggests a starting point.
"If it's going to be anything, [regulation] should be federal. It should be light touch," Jamie Dimon said recently. The JPMorgan Chase & Co. chairman and CEO was wary of fragmented, multi-level oversight: “It’s almost impossible to deal with commerce when you have different state laws for everything.”
Dimon’s perspective is not only that of a top bank CEO, but also of a major deployer of AI. JPMorgan was a launch partner in Project Glasswing, the Anthropic initiative to understand and control newly emerging cyber vulnerabilities.
Jamie Dimon: Go “light touch.”
The JPM chief and his peers, in sectors ranging from financial services to energy to healthcare, deal with legal and compliance complexity both domestically and globally. They may hold out hope that AI can get more rational treatment – if the many points of view can be sorted out, and with some consensus on standards, ideally internationally.
One body of opinion holds that existing rules can apply. “Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products,” former Federal Trade Commission Chair Lina Khan has written, “We shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books.”
Currently chaired by Andrew Ferguson, the FTC has reportedly opened a probe into whether developers including Anthropic and OpenAI are causing consumer harm.
The European Union’s AI Act was touted as a template for regulatory protections. Taking effect in 2024, the law predated, and therefore was not influenced by, the recent breakouts of agentic models from Anthropic, OpenAI and others. These gave rise to apocalyptic predictions – as well as U.S. federal and state legislative proposals variously calling for registration or vetting of frontier models, mandatory disclosures about models and their safeguards, and kill switches for out-of-control bots.
The Trump administration, after an initial clampdown following Anthropic’s April release of Claude Mythos, has generally opposed restrictions on AI development – and on data center construction – so as not to impair U.S. technological competitiveness.
States Take Action
California Governor Gavin Newsom signed a bill September 9 which he trumpeted as “nation-leading action to advance AI safety, transparency, accountability, and responsible innovation, but the scale and potential consequences of this technology demand sustained action from every level of government. The federal government must step forward with robust, national regulations that match the urgency of this moment.”
In a September 23 letter, 26 state attorneys general, including California’s Rob Bonta and led by New York’s Letitia James, urged congressional leaders of both parties “to quickly pass substantive legislation to regulate the AI industry to ensure that development occurs at an intentional pace, incorporates safety and transparency by design, and maintains states’ ability to oversee the industry.”
“In recent weeks, alarming reports of AI agents breaking containment have shocked the nation,” said James. “My colleagues and I are calling on Congress to act swiftly to establish a regulatory framework for AI development to ensure this does not continue.”
New York Governor Kathy Hochul signed the Responsible AI Safety and Education (RAISE) Act, imposing on the AI labs transparency, safety and incident-reporting standards. It assigned a significant implementation role to a new Office of Digital Innovation, Governance, Integrity and Trust (DIGIT) within the New York State Department of Financial Services (DFS).
This is not the first time financial regulation has entered into the high-tech conversation – as a solution or framework.
Systemic Big Techs?
Anticipating a blurring of financial industry boundaries as Big Tech companies such as Amazon and Google diversified and accumulated market power, Fernando Restoy of the Bank for International Settlements’ Financial Stability Institute advocated in a 2021 paper “level playing field” and “same activity, same regulation” policy principles.
“As things move forward, and some Big Techs continue to increase their presence in the financial services market, their operations may acquire systemic importance,” Restoy wrote. “This should be acknowledged by the regulatory framework. A complication is that they operate across a range of financial and non-financial business lines, thus requiring cooperation across different authorities.”
Fernando Restoy
He added that “the potential of Big Techs to achieve a dominant position and to use that position to adopt anti-competitive practices may deserve specific action. An entity-based regulation targeting those risks, including rules that facilitate comprehensive and efficient data-sharing, seems a promising strategy.”
With a nod to the financial regulatory designation of systemically important banks, the U.S. Cyberspace Solarium Commission in its 2020 report recommended a systemically important entity (SIE) designation for a broader set of critical infrastructures. (The congressionally mandated commission closed in 2021 but continues as a nonprofit tracking some of its recommendations.)
Self-Regulatory Model
AI luminaries Demis Hassabis and Dario Amodei, in lengthy essays, took cues from financial regulatory playbooks.
“There is both huge excitement and uncertainty around AI, and both are warranted,” according to Hassabis, co-founder and chairman of Google DeepMind and chief scientist of parent Alphabet. “But the future is not yet written; we must use this precious window before AGI [artificial general intelligence] arrives to shape this technology for the benefit of all’ humanity.”
Demis Hassabis
In a section of his July X post headed “A Framework for a Frontier AI Standards Body,” 2024 Nobel Chemistry laureate Hassabis wrote: “The rapid progress we’re seeing in AI requires a new approach to testing frontier AI model capabilities that is dynamic, adaptable, and rigorous. The U.S. is well positioned, given its economic and technical standing, to take the first step in developing such a framework.
“It could establish a new Standards Body modeled on a federally overseen public-private partnership or self-regulatory organization, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives. Funding would need to be substantial and likely mostly come from industry, in order to attract world-class technical talent and provide the necessary compute resources for large-scale testing.”
FINRA operates independently under Securities and Exchange Commission supervision and is funded by fees from its member broker-dealers.
“Pacing the Frontier”
Anthropic co-founder and CEO Amodei’s 3,800-word blog on September 10 made headlines asserting, “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain . . . Left unchecked, [AI] could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”
Dario Amodei
Amodei mapped out a three-step plan for “pacing the frontier”: embedded third-party evaluators; democratic coordination among frontier AI companies on common safety standards; and global coordination in which “democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.”
The role of embedded evaluators, Amodei explained, “is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory ‘supervisors’ embedded along with employees.
“Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.”
Critiques and Questions
“The regulatory backdrop into which all of this lands is unsettled,” noted a Cloud Security Alliance (CSA) analysis of Hassabis. “A June 2026 congressional discussion draft, the Great American Artificial Intelligence Act, would preempt state AI development laws for three years while codifying CAISSI’s [NIST Center for Advancing Innovation and Standards for Super Intelligence] evaluation role and imposing binding disclosure requirements on ‘large frontier developers’ with more than $500 million in annual revenue.” An earlier 99-to-1 Senate vote “stripped a 10-year state-law moratorium from other legislation, illustrating how contested federal preemption of state AI authority remains.
“Hassabis’s proposal does not resolve that federal-versus-state fight; it instead proposes a third structure, industry self-regulation under loose federal oversight, that could operate alongside, ahead of, or in tension with whatever Congress eventually enacts.”
The CSA saw as a core security question “not whether frontier models should be tested before release, a point on which there appears to be little remaining public disagreement among labs, government evaluators, and outside researchers, but whether the testing regime itself can be trusted to produce evaluations that mean what they claim to mean.”
The alliance regarded as a “second, structural security concern” such aspects of self-regulation as the “issuer-pays” model of funding the evaluator, whether tech vendors have “the capacity for credible self-regulation,” and inherent conflicts of interest.
Sebastian Mallaby
Sebastian Mallaby, a Council on Foreign Relations senior fellow and author of “The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence,” in a New York Times commentary connected Hassabis and Amodei. The latter “is proposing evaluators because Congress is unlikely to act quickly to create a government regulator that forces labs to act responsibly – the most obvious route to coordinated pacing. But he only briefly mentions another coordination mechanism that might prove useful: an industry-financed but government-endorsed self-regulatory body” as proposed by Hassabis,
A conundrum of any pacing slowdown is “how to buy more time for safety without falling behind China,” which is believed to be trailing in “the race” by only a few months.
“The icy state of U.S.-China relations is what makes an AI slowdown so elusive,” Mallaby concluded (before the September 23-25 Trump-Xi summit). “The technology’s positive potential will be realized only if the Trump administration and tech leaders like Mr. Amodei throw their full weight behind AI talks with Beijing.”
Commissions and Boards
Among other regulatory ideas:
-- Harvard Law School professor Cass Sunstein believes an AI Regulatory Commission is an urgent necessity “because of current risks and because of growing public concerns.” His “incomplete and tentative” outline on Substack includes rulemaking and investigatory powers and possible funding by AI companies.
Cass Sunstein
Sunstein acknowledges that the Supreme Court “has recently thrown the whole idea of independent agencies into the constitutional garbage heap, with an exception for the Federal Reserve Board.” Nonetheless, “there is a good argument, in principle, that the AIRC should be an independent agency, to reduce political pressures in this domain.”
-- In The World Needs an AI Stability Board, Paul Samson, president of the Centre for International Governance Innovation (CIGI) in Canada, envisions “a coalition of aligned powers moving to establish a new institutional structure . . . a government-led standing forum for governments, companies and independent experts to build international cooperation on AI.” Samson’s model is the G20-endorsed Financial Stability Board, which “has effectively brought together existing national and international authorities to assess systemic vulnerabilities, improve coordination, and encourage common standards and best practices — with legally non-binding decisions. There are many parallels to the kinds of coordination issues faced with AI today, and the warnings of the need to act quickly to fill the current gap in governance.”
-- In The White House has an AI oversight plan. But who will do the overseeing?, Lauren Kim of the Atlantic Council’s GeoTech Center makes a case for “an oversight board that regulates a third-party auditing market.” This would be “consistent with the Trump administration’s stated concern about avoiding overly burdensome regulation,” and “modeled after regulatory structures created in financial auditing,” namely the Public Company Accounting Oversight Board (PCAOB). “The Artificial Intelligence Auditing Oversight Board (AIAOB), as it could be called, would balance governmental oversight with a third-party auditing market,” register and license audi\ting firms, and be funded by their licensing fees.
Bronwyn Howell, an American Enterprise Institute nonresident senior fellow, doesn’t put much stock in a slowdown of AI model development or in “relying on governments – local, federal, or global – to coordinate and regulate any activities in the current geopolitical climate.”
Jensen Huang
Inability to “coordinate a worldwide AI development slowdown does not mean accepting AI Armageddon as inevitable,” Howell has written. “The knowledge of how the tools work, and how to manage and govern them, lies within the firms themselves, not governments . . . AI leaders should look within for solutions rather than kicking the regulatory can down the road to the politicians.”
GPU chip giant Nvidia is making that kind of effort with the Open Agent Safety Platform, joined by a host of “leaders from across the AI ecosystem [including Anthropic, Citi, JPMorgan, Microsoft, Palantir and Salesforce] to strengthen AI safety for every industry across the full stack of infrastructure, software, models and robotics.”
“AI’s extraordinary potential for society will only be realized if we solve AI safety,” Nvidia CEO Jensen Huang said in the announcement on September 28, the day before he and other top AI company executives attended a White House meeting and discussed “self-policing.”
“As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety," Huang went on. The Nvidia platform “brings together industry, researchers and public-sector organizations to share best practices, align on evaluation methods and foster international cooperation. Together, we can raise the bar for global AI safety.”
Topics: Cybersecurity, Regulation & Compliance, Model Risk, Data & AI Model Governance