If artificial intelligence is prone to errors or hallucinations, what is to be done when they cause a medical misdiagnosis? A patient’s suffering aside, who bears – or covers – the potential liability of an errant chatbot?
Diagnostic errors are a leading cause of “preventable harm” in primary and emergency healthcare settings, but as a risk case, it can be illustrative of any number of AI applications in finance or other business sectors. Accompanying the vaunted benefits of generative AI, Carlos Rodriguez Sanz and Stéphane Vauterin of AXA XL have written, are multiple challenges including data security vulnerabilities, regulatory concerns and reputational risks.
“As a result, a critical issue facing nearly all large companies today is ensuring that the power of AI is harnessed responsibly and sustainably,” says their April 2025 article. They outlined “three broad categories of threats for companies to consider as they integrate different AI-powered tools into their operations”: adversarial threats from malicious actors; situational or internal threats when AI is incorporated into business operations; and external and environmental threats, including systemic risks.
“Businesses that depend too heavily on AI may suffer systemic failures if models malfunction or produce inaccurate results,” for instance. For another: “Many businesses rely on a few AI providers, leading to digital monocultures that are less resilient and more susceptible to catastrophic cyber incidents.”
For those types of exposures, insurers are enhancing cyber risk policies; AXA XL introduced an endorsement to its CyberRiskConnect policy for generative AI, with, for example, liability coverage for breaches of regulations such as the European Union’s AI Act.
Specialty insurer Relm “provides crafted coverage to meet the demands of AI innovators, empowering businesses to scale responsibly, manage exposures, and operate securely in one of the fastest-growing sectors of our generation,” according to its website. A suite of products are designed “to provide clarity and confidence in an area where traditional approaches fall short, enabling businesses to innovate with AI securely and responsibly, CEO Joseph Ziolkowski said when they were introduced.
Thus cyber exposures and the need for adequate coverage have filtered up to the corporate board level in the context of fiduciary duty.
Richik Sarkar and Jarman J. Smith of the law firm Dinsmore and Shohl, in Risk Management Magazine, warn against “blind overreliance on AI . . . Companies should not only prioritize AI models that aid in risk analysis and liability determination, but that are also transparent and explainable.
“Traditional insurance products may soon become outdated if companies do not review and update them regularly. In the age of AI, substantial gaps in current coverage exist. For example, human error remains a driving cause of cyber risks, and many insurance policies do not cover incidents like fraudulent fund transfers that AI may worsen if improperly trained.”
Addressing AI errors caused by chatbots is Armilla, a startup “AI specialist” insurer with which Lloyd’s of London partnered on a warranty protection product. As reported in the Financial Times, Armilla chief executive Karthik Ramakrishnan said the product could encourage AI adoption by companies that were deterred by fears that tools such as chatbots would break down.
Underlying the protection are Armilla Guaranteed, a contractual performance guarantee for AI vendors, backed by a consortium of reinsurers; and Armilla Insured, liability specifically designed for risks associated with deploying AI.
Steve Morris of Newmedia.com
“It’s both more and less than insurance,” explains Steve Morris, founder and CEO of digital marketing agency Newmedia. “More, because it has the power to hit you where it hurts financially. Less, because it acts like a circuit breaker that alleviates open-ended risks that keep directors awake at night by setting a quantifiable ceiling on potential losses.”
“Most importantly,” he adds, “in my experience it tends to unfreeze legal and compliance teams that have been holding back AI adoption because of that hazardous unknown tail risk.”
The Armilla offering is built on an errors and omissions (E&O) “chassis” with the added warranty component. “Think of it less like traditional insurance and more like a performance guarantee,” says Michael Giuliano of risk solutions firm McGill and Partners. If a chatbot “operates outside its defined parameters and causes financial loss, the warranty responds with payment. That’s very different from today’s typical E&O structure, where a customer would have to sue the technology provider to access coverage.”
Michael Giuliano: A departure from E&O.
“It’s a novel concept [that] recognizes that AI introduces unique risks traditional policies don’t fully address. But it’s still early, especially in healthcare,” says Jim Olsen, chief technology officer of ModelOp.
He suggests that who is legally liable or accountable needs to be clarified: Is it the model or chatbot vendor, the company implementing the chatbot, or a user relying on the information it produces?
The lack of legal certainty was pointed out in a New York Times article about a suicidal teenager who sought advice from ChatGPT. Eric Goldman, co-director of the High Tech Law Institute at the Santa Clara University School of Law, said, “There are so many questions about the liability of internet services for contributing to people’s self-harm. And the law just doesn’t have an answer to those questions yet.”
In The Risks of AI-Specific Liability Regimes, American Enterprise Institute nonresident senior fellow Daniel Lyons critiqued the European AI Act – which inspired a number of U.S. state laws – for “wrongly treat[ing] the tool as uniquely suspect, rather than focusing on the harm caused.”
Morris says he has observed “a distinct uptick in demand” across multiple industries for “AI guarantees with teeth. What’s driving it is actual liability, not ‘AI hallucinates’ clickbait.” He also notes interest in AI compliance audits centered on dialogue, as opposed to coding.
Jim Olsen of ModelOp.
“We’re entering a feedback loop of accelerating adoption in highly regulated industries,” Morris adds. “More than half of financial services executives expect a catastrophic incident in the next 12 months due to generative AI. Roughly 45% of enterprise execs see reputational risk as a major barrier to adoption of generative AI” – findings from Deloitte’s Generative AI Dossier.
While hesitant to call it a full-out trend, Olsen says awareness of AI-error exposure is elevated due to publicized incidents such as Air Canada’s liability for its chatbot’s misinformation. The ModelOp CTO sees a difference between relatively low-risk, consumer-facing failures and healthcare or financial services where the stakes are patient well-being and portfolio holdings.
Over time, however, the AI warranty concept may evolve from niche-differentiator to table stakes, just as cyber insurance did a decade ago, according to data and AI strategist Vinod Goje. Competitive pressures will force vendors and enterprises to act, and there will be options for transferring or off-loading the risk.
Morris sees a near-term “rise of quantitative vendor diligence. Such products already shift the burden of technical diligence into risk teams, in much more granular form than just proving ‘best efforts’ or using checklists. These warranties represent a hook that gives risk managers a quantifiable point to place on a register, and a safe harbor bounded by a contract. That doesn’t mean buyers can let their guard down, though.”