Rethinking Technology Risk: Moving Beyond Guns, Gates and Guards

Today, “technology risk” extends to everything from, say, artificial intelligence, machine learning, blockchain and cybersecurity to data, digital currencies and cloud computing. That’s a stark difference from 50 years ago, when weapons of war and biotechnology were essentially the only areas that received this designation.

Indeed, back then, the dangers from computers, AI and robots were strictly for science fiction – and even nuclear technology risk did not get a lot of attention. But while technology has grown explosively and while risk management is not as narrowly focused as it once was, it’s fair to question exactly how much progress has been made.

Are we now adequately assessing both the risks and the opportunities of individual events, or do we still rely too much on an old-guard approach that is focused on dangers and defenses? Do we, moreover, fully understand the impact of regulation on risk, the interconnection between humans and the environment, and the need for innovation and risk-taking?

Aaron Brown

Before answering these questions, it’s helpful to wind the clock back and explore how we managed technology threats (particularly biotechnology risk) in the past, as well as the advances we’ve seen in more recent times.

Then and Now

In 1975, the Asilomar Conference on biotechnology risk issued a report that has come to be seen as a classic example of the “guns, gates and guards” approach to risk management. The basic approach was to assess the dangers — not risks — of individual projects and to implement an appropriate set of biological and physical defenses. (On the biological side, one could express, for instance, the recombinant gene in an organism that cannot survive or reproduce in the wild, while stringent containment inside a laboratory would be an example of a physical defense.)

Modern financial risk management grew out of dissatisfaction with traditional approaches like this. They are danger-minimization rather than risk management. They can work only for anticipated risks, and usually fail even for those, because they don’t allow for human and natural reactions to evade restrictions.

Last year, the Council on Foreign Relations held a workshop — “Managing the Risks of Biotechnology Innovation” — to reassess the issues raised in 1975. Its first conclusion was that it is impossible “to wall off biotechnologies to prevent their misuse or to hold information related to biotechnology secrets.” The guns, gates and guards strategy, in other words, is not enough even to contain biotechnology dangers.

The CFR’s second conclusion also reflects a more modern view of risk management. It stated, in short, that we must manage the opportunities of biotechnology, as well as the dangers, because the benefits of this field would not just naturally evolve.

When we take a closer look at the evolution of technology risk, including biotechnology threats, we can see that there are still blind spots in traditional thinking about risk — thinking that remains ubiquitous outside of the professional risk management community and a few thoughtful observers.

Flaws Remain

Only three years after the Asilomar conference, the guns, gates and guards approach failed in spectacular fashion.

The last natural case of smallpox in the world was recorded in 1977 in Somalia, and the only remaining viruses were in laboratories held under the strictest possible containment. Unfortunately, in 1978, photographer Janet Parker came down with smallpox, and died within weeks, at Birmingham Medical School, where some of the smallpox pathogens were held. Roughly 300 people were quarantined at BMS, but fortunately there were only two deaths: Parker and the lab director, who committed suicide.

Currently, the U.S. lists 68 “select agents and toxins” that pose exceptional risks. They are held worldwide in 234 registered laboratories with 8,516 approved individuals, according to the CDC's 2022 annual report. Those numbers, however, do not account for an unknowable number of unregistered facilities and unapproved individuals with access. Moreover, regulations do not cover attenuated strains and inactive forms, which in the past have turned out to be less attenuated and more active than expected.

It is no surprise that with so many people handling dangerous agents, there were 170 reports of accidental release in 2022 — the majority from entities that were not registered. Nobody died as a result, but 595 individuals were exposed. What’s more, one can imagine the reported releases do not represent the true worldwide total, including secret and illegal facilities.

But the problem is not only with illicit or small facilities. In 2014, the U.S. Food and Drug Administration came across hundreds of unregistered virus samples — including smallpox — in a cardboard box in its offices that may have been there since the 1960s. Only one vial had released its contents, and it was not a deadly virus — but that was pure luck.

Important Lessons for Risk Managers

The fact that guns, gates and guards approach is seldom as efficient as designed is the minor problem. The major problem is containing known dangers is much too narrow an approach to risk management. There are three additional key takeaways from the 2023 CFR conference:

1. Pathogens cannot be locked away. Most deadly pathogens, including Bacillus anthracis, the causative agent of anthrax disease, and Francisella tularensis, which causes tularemia, are found in the wild. Furthermore, the genetic material encoding pathogens can be chemically synthesized (or ordered from a company that specializes in synthesizing long stretches of genetic material), and the genetic code can be “booted up” in a laboratory. Theoretically, there is no limit to any pathogen being laboratory created in this fashion.
   
   
2. Life will find a way. Biotechnology risk is not limited to the effect of pathogens on humans. Risk management must recognize the interconnection between people, animals, plants and their shared environment.
   
   In biology, this is often called the “one health” approach. This affects our perception of dangers — a plague that “just kills animals” or a physical environmental change can do as much or more damage as a direct attack on people. On the other hand, these dangers can also open up opportunities. Vaccinating animals or improving ecosystems, for example, could potentially aid human health and comfort.

3. Money will find a way. Regulations must be understood in the context of how they change financial incentives. We need good innovation and risk-taking to be profitable, and bad innovation and risk-taking to be unprofitable. As long as there are contrary financial incentives, simply mandating the good and outlawing the bad will fail. 

These are lessons for all risk managers, not just biotechnology risk managers. Yes, where appropriate, it’s wise to have a good guns, gates and guards approach. But that should be only one tool of danger minimization, rather than an all-encompassing risk management strategy.

The things being guarded will creep out, somehow or other. Maybe not soon or all the time, but enough that we need contingency plans when information or ideas or other things leak, despite our best efforts.

Risk managers should have great respect for evolution, both in the natural world and among human institutions. Think of the big picture and all the interconnections — how squeezing down risk in one place often leads to worse risk, and unmonitored risk, in other places.

Parting Thoughts

You don’t have to be an expert in biology to see the lessons of the last 50 years of biotechnology — its incredible benefits and its disasters. That history has led to excessive optimism and bubbles — both financial and ideological — and Luddite popular opposition.

Risk management outside of finance has remained mostly old-fashioned and inadequate. Ultimately, it must change. Lessons from biotechnology history can inform risk managers today, helping to bring about much-needed revisions.

The way we view profit and risk is one necessary change. Profit incentives should be used as a tailwind to speed your risk management success, rather than something to overcome.

Remember, good risk management is a complete culture, not just a set of principles and procedures. It should cover customers, suppliers, regulators and the general public, as well as institutions and their shareholders.

Aaron Brown worked on Wall Street since the early 1980s as a trader, portfolio manager, head of mortgage securities and risk manager for several global financial institutions. Most recently he served for 10 years as chief risk officer of the large hedge fund AQR Capital Management. He was named the 2011 GARP Risk Manager of the Year. His books on risk management include The Poker Face of Wall Street, Red-Blooded Risk, Financial Risk Management for Dummies and A World of Chance (with Reuven and Gabriel Brenner). He currently teaches finance and mathematics as an adjunct and writes columns for Bloomberg.