A Defense of Empirical Risk Management

Is risk modeling broken? Many so-called experts are now beating that drum after flaws in historical-data-driven models were exposed amid the pandemic. Some are even saying we should completely ditch empirical methods in favor of a more qualitative, judgment-based approach.  

This leads to the million-dollar question: do traditional models still have value, particularly with respect to loss forecasting? To find out, we have to peel back a few layers of the onion.

It is true that most stress testing models failed during the pandemic. It was not that credit losses were predicted to be low, meaning that banks faced ruin by relying on overly optimistic prognostications. Rather, the models predicted a wipeout due to a severe, lockdown-induced spike in unemployment. In the end, though, credit losses barely whimpered, with some key metrics even improving as the pandemic unfolded.

This was the second recession in a row where the industry’s models, viewed generally, were perceived to fail. During the subprime crisis, most mortgage-loss projections used by banks were found to be severely underweight. 

As the dust clears from COVID, many risk practitioners have concluded that the very practice of risk modeling is ruptured. To be clear, they are not saying that the specific cohort of models needs to be respecified, but that the very notion that models are useful tools for risk management is now outdated. 

They have decided that qualitative, narrative-based methods – ostensibly the scenario analyses that were developed during the grim months of the global financial crisis – eat models that rely on historical data for breakfast.

There is no doubt that the intelligence provided by statistical techniques is restricted to the domain of the observed data. Such methods can, though, identify groundswells in history, extrapolate their effects and thus predict future outcomes that are quite distinct from past behavior.

The domain of the human imagination, meanwhile, is boundless.

Reimagining the Past

Before delving further into the usefulness of traditional models, it makes sense to consider whether risk managers would have improved the accuracy of their forecasts if they had taken a more qualitative approach and weighed more extreme downside scenarios, pre-COVID.

Prior to 2020, it would have been quite reasonable to propose that banks consider the possible impact of a global pandemic. How might this have worked out?

An empirical approach to the problem would have involved examining the effects of past pandemics (e.g., the 1918 influenza outbreak and the more recent SARS and MERS events), and to then use these as the basis for inference about the potential impacts of a possible new outbreak.

None of these proxies would have been a perfect fit for COVID, but they might have suggested that people would be willing to comply with collective containment strategies. Moreover, they would have offered clues about the nature and strength of government response. However, I still doubt that such reasoning would have generated accurate credit-loss predictions prior to the 2020 pandemic.

Tony Hughes

Even with foreknowledge, risk managers weren’t just going to sit back and offer optimistic views on future defaults as unemployment was spiking amid a crisis. No-one, after all, would have taken such projections seriously. (An important aside: it’s easy to imagine that every potential crisis will kill every bank, but history shows us that bank portfolios are often incredibly robust, especially in the face of crises whose genesis lies outside the banking industry.)

Beyond hypothesizing about government response to COVID, it would have been possible to imagine scenarios where societal institutions broke down, with outbreaks of crime and violence as desperate people fought for resources in a life-or-death struggle. Scenarios like this, involving the collapse of governments and economic devastation, could not have easily been dismissed a priori. Such outcomes, while unlikely, would have been very damaging to bank financial health.

But what would a banking executive do when presented with such musings? What adjustments in strategy would they prompt? If such an outcome was certain, would it even be possible for a bank to prepare effectively? If the organization happened to jump at these particular shadows, there’s every reason to believe its odds of survival would only be harmed.

Let me elaborate. Before a potential crisis, when the situation is stable but people are fretting about a looming threat, an active response means diverting the bank away from its previously determined optimal strategy. There are four possible outcomes:

1. The threat does not occur;
2. The threat occurs but is either irrelevant or advantageous for the bank (this happened with COVID), given its original strategy;
3. The threat occurs, harms the bank’s performance and the change in strategy is effective in mitigating loss; or
4. The threat occurs and harms the bank’s performance but the change in strategy is ineffective.

 It follows that the change in strategy harms the bank’s performance in three of these situations. In the fourth, the bank’s performance is harmed prior to the onset of the crisis, after which the new strategy becomes a savior.

With the benefit of hindsight, the most consequential preparations in advance of COVID were those designed to ensure business continuity in the face of a generic disruption. Over the past decade, banks have been put in place operational arrangements – like redundancies in IT systems and interminable work-from-home drills – to allow servicing of clients across a wide range of possible contingencies. These arrangements would have been effective for a number of unlikely but highly problematic disruptions – not just for the pandemic but also for events like general transport strikes, nuclear accidents and even small wars.

 Learning from Experience

Looking forward, the key question concerns the best use of a generic risk manager’s time.

History may not provide the full range of possible happenings, but the signals it does provide are generally reliable and useful. Scenarios that are the product of human imagination are unbounded in scope, but the quality of speculative analysis for very unlikely tail-risk events cannot be discerned. It is therefore difficult to imagine a prudent manager making consequential, fork-in-the-road decisions on the basis of such musings.

So, while the 2019 cohort of risk models is clearly in need of an update, the principles of empirical analysis should never be abandoned. We need to fully process the lessons of history – statistics and modeling are tried-and-true methods of pursuing this aim efficiently.

The fact that we have to update our models following COVID is not a sign of weakness but one of strength. The old models were found to be poor, but now they can be superseded. We’ve learned a lot about our portfolios through the pandemic; it would be odd if these findings were not reflected in the models we are using to predict future performance.

The overarching lesson here is that risk management needs to remain disciplined. Empirical approaches – learning from experience, if you will – provide a conduit through which this can be achieved.

Scenario analysis, in contrast, offers too much scope to examine questions that are not relevant to decision making. Furthermore, the fact that scenarios cannot be effectively validated is a genuine shortcoming that severely limits the usefulness of the technique when applied at the coalface.

Don’t get me wrong: the world is a strange place, and the future will invariably throw up challenges, like COVID, that are well outside the range of our experience. Dreaming of these possibilities in advance, though, is not a disciplined use of our time as risk managers.

The best we can do is to ensure that our businesses are fleet of foot and able to adapt to a rapidly changing external environment.

Tony Hughes is an expert risk modeler. He has more than 20 years of experience as a senior risk professional in North America, Europe and Australia, specializing in model risk management, model build/validation and quantitative climate risk solutions.