dataAs jurisdictions around the world move to regulate artificial intelligence and the industry’s biggest names try to have their say, a new venture is offering to bring quantitative objectivity to organizations’ governance of the technology.
The AIQ Score, administered by AIQA Global, takes into account five aspects of AI implementation: strategic alignment; oversight and accountability; technical robustness; responsible AI and compliance; and adaptability and education. The score ranges from 0 to 200 based on 250 data points across those dimensions.
“The methodology is grounded in recognized governmental and non-governmental standards, including the NIST AI Risk Management Framework, the EU AI Act, and ISO/IEC 42001, the Chicago-based AIQA said in its April launch announcement.
“Every major asset class eventually gets a quality rating,” commented AIQA’s chairman, James E. Malackowski. “Credit quality has its rating agencies. Patent portfolios got Ocean Tomo Ratings. AI governance now has the AIQ Score.”
James Malackowski
Malackowski co-founded Ocean Tomo and is chief intellectual property officer of its parent, J.S. Held. He created the Ocean Tomo 300 Patent Index (NYSE: OTPAT) and the OTR PatentRatings system.
He co-founded AIQA with his son, Chase J. Malackowski, who developed the 250-factor AIQ methodology. Chase is the firm’s managing director, chief technology officer and head of product, and a Georgetown University graduate student in AI Management.
Numerous public- and private-sector initiatives and discussions have centered on AI safety and standards, strategic alignment, quality of implementation and risk management. In a March GARP Risk Insights article, Michael Mainelli and Maury Shenk posited that there are limits to what “top-down, outside-in” regulation and standardization can achieve. Advocating “working from the inside out,” they co-founded Ordinary Wisdom to build infrastructure for “value-aligned systems”; their scope includes financial services use cases.
Shenk, Ordinary Wisdom’s CEO, said it appeared that AIQ Score could be compatible with the evaluation criteria of his new organization, which is focused on model behavior.
In a similar vein, ImmuniWeb, which provides an AI-based platform for application security, is offering a free online CyberScore tool for “all internet users to run a quick, simple and actionable cybersecurity due diligence on themselves, their external stakeholders and business partners,” said CEO and chief architect Ilia Kolochenko.
ImmuniWeb says it assesses AI readiness alongside traditional cybersecurity indicators; bases the rating on such metrics as website security, encryption security and dark web exposure; and will make the CyberScore “available to ImmuniWeb Discovery customers as a third-party risk management (TPRM) module.”
Also at the intersection of Ai and cybersecurity, AIQA entered into an agreement with SecureSky to “pursue co-authored thought leadership and coordinated client education programs. The arrangement gives AIQA a trusted resource to which it can refer clients whose AI governance assessments surface cybersecurity exposures, and gives SecureSky a recognized independent authority on AI governance quality for its clients deploying AI at enterprise scale.”
AIQA’s founders cite trends such as litigation risk as an incentive for companies to demonstrate quality governance. “Self-attestation is not governance,” as James Malackowski put it. “Independent measurement is.”
Chase Malackowski
J.S. Held’s AI Disputes Monitor was tracking 352 AI-related lawsuits in the U.S. as of April. Paligo analyzed over 1,400 harmful AI incidents, finding that 49% “involve software-only systems, including cases where AI-generated content errors in publishing and documentation caused direct financial and legal consequences.”
In view of rapid technological advances and new issues constantly emerging, the AIQ Score does not seek to capture whether a company has “the latest and greatest AI technology,” Chase Malackowski pointed out. “We’re trying to grade on how well you position your firm or business to be ready to adopt AI as it changes.
“By adopting strong AI governance, strong AI documentation and processes, you will be ready to adopt the next system when it is released, and you’re not going to be playing catch-up.”
The score’s 250 quantitative data points are divided into five groups of 50 each, Chase explained. The scoring is similar to an IQ, with a 200 maximum and 100 mean.
For comparability, “you are not as a healthcare provider going to be compared to an insurance company. Any size company in your vertical should have comparable scoring methodology. So whether your company has 10 employees or 10,000, the scoring method should work across the whole industry.”
The scoring criteria and methods are transparent, in part to ensure consistency over time, the AIQA managing director went on: “There will be no black box system that determines any of these ratings, and they are all reproducible. At any point, you could go back and use the same model to retroactively grade an enterprise with their systems that were in place two years ago, and the rating should be replicated.”
Vedder was the first law firm to sign on for AIQ Score. The Chicago-based firm “not only wanted to have the benefit of the guidance, but they wanted a way to communicate to their clients that AI is a well-governed mechanism within the firm, which just speaks to the whole mission of having that independent voice that companies can use for their stakeholders,” James Malackowski said.
The AIQ Score often highlights a common risk that Chase described as prompt and dataset drift: “We want to ensure that the systems that companies are using, their dataset, are properly governed, properly protected and properly tracked and verified, so that there’s not going to be major output drifts through their AI systems. As a lot of people in the AI industry will say, the output is only as good as the input they can start with.” Dataset imperfections “is something that we want to continuously track and ensure doesn’t happen.”
Companies educating staff about AI can reduce their risks and raise their score.
Said Chase, “Is your firm or organization properly educating its users of AI tools on the impact that AI will have on their work? Maybe they’re concerned this is going to take their jobs, but this is going to simplify their life.”
Oversight should extend to use of third-party AI tools. “Make sure they have the proper governance in place that you review, you check that they’re trustworthy before you interject them into your own system and then have whatever problems that may create.”
Chase Malackowski envisions AIQ scores functioning like an accreditation, and AIQA eventually having at least “shadow” ratings for every publicly traded company as a service to investors. These ratings would be based on public disclosures, with AIQA applying models using its own data.
In late May, AIQA Global announced publication of the Chicago Principles for Independent AI Assurance, designed to simplify the firm’s collected knowledge in informing policymakers. “We are in the process of educating legislators on what some of the absolute core of our principles may need to be, and we think that’s a contribution to the industry,” James Malackowski stated.
While there are established frameworks for “what good AI governance looks like and what standards organizations should meet,” according to AIQA Global, “none specifies the conditions under which third-party verification of compliance with those expectations is itself trustworthy. The Chicago Principles fill that gap.”
Six principles are outlined: Independent, Measurable, Auditable, Comparable, Continuously Updated, and Accountable.
The principles “describe what assurance must be to deserve the trust markets are now asked to place in it,” the senior Malackowski said in the release. “They are not AIQA’s competitive moat. They are the conditions under which any provider should be evaluated by anyone relying on the result.”
Another proposal along these lines is the Prosocial AI Index, designed to make “principle-oriented baselines usable for policy makers, boards, regulators and public agencies.” Described by Cornelia C. Walther, a senior fellow of Canada’s Centre for International Governance Innovation, as a “policy-maker-friendly scoring rule,” the prosocial index “translates governance duties into a 16-cell dashboard using color ratings: four Ts (Tailored, Trained, Tested and Targeted) read against four Ps (Purpose, People, Profit and Planet).”
“The existing policy baseline is complex, and incomplete,” Walther wrote, referring to the Organization for Economic Cooperation and Development AI Principles, the NIST framework, ISO/ISE 42001 and other examples. “These approaches may suggest that a policy framework is in place, but the more important question is how that framework can be translated into traceable metrics.”
Jeffrey Kutler of GARP contributed reporting for this article.