Stress Testing in War and Peace: How to be Better Prepared for Future Crises

In 2020, for the first time in 11 years, a U.S. stress test was conducted while the economy was actually in recession. The test has been criticized by detractors who have argued that it did not take full account of COVID-19 scenarios and therefore should have been postponed. But are these concerns valid?

The results of the Fed's modified 2020 CCAR exercise indicate that all major banks are appropriately capitalized, though some would approach minimum levels under the more extreme U- and W-shaped recovery scenarios posited by the Fed.

Still, the exercise attracted considerable disparagement, most interestingly from Daniel Tarullo - the Fed's stress testing czar until 2017 and a key architect of the CCAR process. The criticism offered is largely on point but notable for what it ignores. Tarullo excoriates the most recent test, conducted under war-like conditions, but has little to say about the way peace-time stress testing developed during the recession-free 2010s.

The core elements of CCAR, the standard adverse and severely adverse scenarios, were retained in full in the 2020 exercise. Tarullo's disapproval is not aimed at those projections, but, rather, centered on the role and prominence of the three coronavirus scenarios inserted by Fed overseers a few weeks into this year's process.

In Tarullo's view, the CCAR should have been delayed, so that the virus-impacted scenarios could take center stage and be subjected to a full analytical treatment. He has considerable faith in the ability of banks to model the current crisis right now.

The most consequential element of the critique was that capital disbursements should be suspended until the path to recovery is clearer. Given that we are beholden to a mindless virus, and that a lot of time and resources have been allocated to research programs bent on its control or eradication, this seems prudent.

However, war-time stress testing is hell, and it's therefore hard to be too critical of the Fed's recent actions. Moreover, considering the inadequacy of current COVID-19 data, it's also fair to question whether Tarullo's confidence in banks' existing modeling capabilities is misplaced.

Global Financial Crisis vs. COVID-19: Different Circumstances

To fully unpack Tarullo's criticism, we need a little history. The 2009 SCAP stress test was conducted within a few months of the Lehman failure, amid financial market turmoil and economic devastation. Authorities needed a mechanism to restore trust between counterparties, boost liquidity and kickstart the nascent recovery. Though the SCAP was hastily executed, it was perhaps the single most successful policy intervention of the 2008/09 Global Financial Crisis.

I doubt whether anyone has since reflected on the accuracy of the projections made during the darkest days of recession or on the soundness of models used to calculate capital requirements. So, SCAP did not succeed because of analytical rigor.

Rather, it worked because it imposed order out of chaos - it was the right gesture made at precisely the right time. If the underlying numbers were out by 5% - or 500% - it didn't really matter. The important thing was that regulators had loss projections and were clearly willing to brandish them.

The circumstances today are rather different. The market turmoil of 2008/09 is largely absent, though the economic devastation is not. There have been zero failures of systemically important financial institutions. This situation may change with a new surge of infections, but there exist no proximate reasons for regulators to panic. The continuity provided by the standard scenarios is well worth preserving.

That said, one of the few advantages of stress testing in a recession is that you don't need to speculate about the nature of the best scenario; instead, you just need to simulate a range of severities. It therefore makes sense to update the economic projections at the earliest convenience. On this score, Tarullo's assessment is absolutely correct.

His error, though, is to have too much faith in the ability of models to capture the linkages between COVID-19-era credit performance and the economy using data that's currently available. A detailed analysis, as opposed to the back-of-the-envelope “sensitivity analysis” actually conducted, would have had little impact on the quality of forecasts used in the 2020 CCAR. At the end of the year, consistent with the Fed's timetable, we will have some early COVID-19-impacted performance data with which to conduct a proper analysis.

If there are weaknesses in the current process, these should have been addressed during the long economic expansion. The whole point of stress testing is to keep the financial sector on a war footing; the time to build the analytical arsenal is undoubtedly during periods of dÉtente.

A Supplementary 'War Games' Solution

So, what was missing on day zero of the COVID-19 crisis?

It would be ludicrous to suggest that every bank should have been required to develop a complete capital plan for a global pandemic by 2019. This was far from the most obvious risk facing the financial sector at the time. The problem, rather, is that no-one built such a plan.

We could all construct top 10 lists of plausible but unlikely events: natural disasters, geo-political risks, cyber threats, climate risk calamities, major economic policy missteps or systemic problems of the banks' own making (like the subprime crisis or leveraged loans). All of these would pose unique challenges that can't be captured by the analysis of a generic, exogenous recession scenario, no matter how severe.

What I would propose is that banks be required to engage in simulated "war games" each year as part of the peace-time stress testing regimen. Regulators would assign a severe event to each bank and ask them to prepare a mock capital plan covering the situation.

Empirical analysis would be based on anonymized bootstrap samples drawn from the bank's own portfolio. Critically, the resulting analysis and documentation would be reviewed by peers and regulators, and then published. More pressing risks could be shuffled between banks each year to establish a range of possible responses to the hypothesized events, viewed from a wide variety of different perspectives.

The core elements of CCAR would be retained - a logical approach, since we have no evidence that capital settings derived from this process are dangerously misaligned. The proposed war games would be supplemental and, importantly, not subject to the heavy governance that characterizes the current CCAR. This will free analysts to explore the assigned risk creatively and fearlessly.

Parting Thoughts

Let's face it, CCAR 2019 was very similar in structure and execution to SCAP 2009. Sure, the available data was far superior and the banks were much better at filling in the forms. But stress testing had become boring and predictable - a straightforward compliance exercise if ever there was one.

If stress testing is only about capital levels, then it's unclear if this is even a problem. Banks, after all, seem solid and well capitalized. This is despite the fact that everyone was blind-sided by the pandemic and its extreme virulence, both actual and economic.

If we are uncomfortable with these kinds of intelligence gaps, there are ways to adjust the operation of the stress test to better prepare us for the next calamity, whatever shape it happens to take.

These measures can wait until after the viral enemy has retreated and peace has finally been restored.

Tony Hughes is a risk modeler and economist. He is a credit risk analytics expert and thought leader with many years of experience building high-caliber risk modeling and data science teams.