Will Increased Complexity Save Scenario Analysis?

Regulators have set their sights on improving stress testing in the wake of the mini banking crisis that unfolded in the U.S. earlier this year. One idea is for banks to consider a wider range of scenarios that consider a multitude of adverse events – but will this really help prevent future crises?

 Tony Hughes

That’s the question du jour. In a recent speech, Michael Barr, the Vice Chair for Supervision of the Board of Governors of the Federal Reserve System, outlined his plan for a series of exploratory scenarios that big banks could apply to broaden the range of adverse events considered by the stress testing process.

What's more, in response to the spring 2023 crisis, central banks around the world have doubled down on the scenario-based stress testing methodology that has been the mainstay of risk management since the global financial crisis (GFC).

Of course, some of the main shortcomings of stress testing came into focus in the wake of the collapses of Silicon Valley Bank, Signature Bank and First Republic earlier this year. The standard stress-testing approach involves trying to assess capital adequacy in the context of a deep, deflationary recession scenario – but that was not germane to the 2023 bank failures, which occurred during a period of solid growth and burgeoning post-COVID inflation.

Consequently, regulators are now planning to rekindle the stress testing process by boosting its complexity. But will this move succeed in making the banking system safer or is there a better approach?

The Complexity Quagmire

As it has matured, stress testing has clearly become more complex. But before we more thoroughly examine the pros and cons of Mr. Barr’s proposal, let’s take a quick look at the evolution of stress testing.

Going back to SCAP, the seminal test that was undertaken while the GFC was raging, banks were asked a very simple question: Do you have enough capital to survive what’s happening outside your window? During early iterations of CCAR, the question changed slightly, with banks being asked ft they had enough capital to keep lending if the GFC recurred?

As the recession faded from memory, this question – which is still ostensibly asked each year – became increasingly distant from the day-to-day troubles facing bank risk managers. Just ask the people recently laid off from SVB.

An interesting way to address this is in the context of Occam’s razor, a core tenet of statistical inference and of life. When faced with a range of alternative explanations, the simplest is likely to be the best. Complexity, according to this tenet, should be avoided unless there is a very clear payoff.

In terms of scenarios, the concept of complexity is a little hard to define. As we've made clear many times, a methodology based primarily on scenarios is not scientific in nature. We can't objectively assess the quality of the work that's undertaken on scenarios, so it’s impossible to measure and difficult to judge the cost of increased complexity.

As the description of a particular scenario increases in detail, our ability to calculate its impact on a specific portfolio should generally improve. If, for example, I ask you what mortgage losses will be during a generic future recession, your answer will start with “it depends” and will cover a wide range of possible outcomes, ranging from “no effect” to “complete disaster.” If the scenario is defined with every possible detail – e.g., the specific customers that will lose their jobs, the winner of the Super Bowl and the future market value of every house in the portfolio – the bank-level projection will be much sharper.

Too Generic

Mr. Barr is not proposing that scenarios be defined with very much increased detail, but, rather, that a range of fairly generic scenarios be employed strictly for exploratory purposes. Since those scenarios are quite general, they’ll provide just a marginal boost to the stress-testing analysis cited in Mr. Barr’s speech.

At low levels of detail, we will need a large quantity of scenarios to cover the full range of possibilities. Moreover, as the number of scenarios increases without limit, the attention we are able to apply to each one will eventually dwindle to zero. If we, for example, run 100 scenarios and something akin to No. 59 actually happens, this is of little value – even if our analysis of the scenario correctly indicated that the bank in question would face difficulties.

Of course, Mr. Barr is cognizant of this issue and is proposing a more modest expansion of the scenarios to be explored. That said, I suspect that diminishing returns from additional scenarios sets in at an extremely low level. It will be hard for a bank, for example, to make itself truly robust against five very different sets of circumstances simultaneously.

The other problem is that this solution is the thin end of the wedge. If Mr. Barr's dissatisfaction with the stress testing exercise is not cured by running more scenarios, and if banks like SVB continue to fail, the Fed would have to go back to the drawing board – but only after facing criticism for its lack of original, productive ideas.

It’s entirely possible, in short, that further increasing the complexity of the range of scenarios considered will only make the stress testing process more unwieldy and inelegant.

Parting Thoughts

Whenever I read regulators describing the wonders of scenario-based supervision, I’m always surprised at the level of faith they have in the value of the exercise. They seem to think that if a strange set of economic events occur and you happen to have the results of a broadly relevant scenario analysis in your back pocket, that everything will be OK. Perhaps they believe that banking crises and failures can only occur when such prior analyses are lacking.

In reality, of course, the projections produced for any given scenario may be wildly inaccurate. We should all remember the poor quality of the loss projections made in the early days of the pandemic and realize that a prior scenario analysis may be a very poor blueprint for the way a bank or regulator should handle an unfolding crisis.

Indeed, once effective risk monitoring procedures have been established and capital is set at a sufficient level, it is difficult to see how additional scenarios will provide a reliable further boost to bank safety.

My other concern is the unwillingness of regulators to expand the range of analytical techniques applied to the stress testing process. We live during a time of unprecedented data richness, combined with the high level of computing power needed to take advantage of it. We could use this data to identify banks, like SVB, whose business models deviate most sharply from those of traditional, conservative banks. We could, moreover, identify clusters of banks that are likely to share the same vulnerabilities and identify products that are exhibiting signs of irrational exuberance.

Instead, Mr. Barr is proposing basically the same thing that was cobbled together in 2009 during a raging crisis – multiplied by five. Contrary to the famous quotation often attributed to Albert Einstein, we can only hope that we can indeed do the same thing over and over again and expect a better result.

Tony Hughes an expert risk modeler. He has more than 20 years of experience as a senior risk professional in North America, Europe and Australia, specializing in model risk management, model build/validation and quantitative climate risk solutions.