A Modeling Reboot: If Not Now, When?

As the world awakens, one of the vexed questions we face as model risk managers is when to redevelop. The simple fact is that the established procedure of designing and building a model, and then having it validated and implemented into various IT processes, is very cumbersome, time consuming and expensive.

Every time we pull the trigger, we do so in the hope that the full procedure won't need to be repeated for a very long time. Thwarting these hopes is the reality that models constantly deteriorate. Even in a perfectly stable, growing economy, all models will eventually need to be rebuilt as consumer preferences, borrower behavior and technology all slowly evolve.

Occasionally, reality can intervene in this process to either precipitate or forestall a looming model rebuild. Following a significant economic shock, you can imagine a prudent model manager opting to wait until the dust settles before initiating a scheduled redevelopment. Conversely, a period of relative calm can provide enough airspace to allow the model overhaul process to be boosted with additional resources.

Scheduled events, like the LIBOR decommission or the introduction of IFRS 9/CECL, can also dictate the timing, by requiring brand new models to be in place well before the circled date in the calendar. Moreover, sometimes, regulators simply demand that models be updated - as the UK's PRA did recently for IFRS 9 models.

Pandemic Impact

Today, a pertinent issue is how COVID-19 will affect the timing of model redevelopment.

When the models missed their target during 2020, some may have argued for models to be rebuilt immediately. One of the necessary but insufficient conditions for a model to be scrapped and rebuilt is that predefined model error tolerances are consistently exceeded. This would have happened for many or most risk models by the middle of last year.

So, let's imagine that a particular bank spent the second half of 2020 redeveloping a business-critical model that performed poorly during the first half of the year. Presumably, the new model would be calibrated to do much better than the original during the early days of COVID-19. As the crisis now lingers, the bank would benefit from having access to sharper, more up-to-date analytics, but there's significant doubt about how long this advantage might last.

If the world snaps out of its virus-induced nightmare, and borrower behavior quickly reverts to its pre-pandemic state, the model rebuild may come to be viewed as a colossal waste of time and effort. It is only if COVID-style behavior is now permanent - and it may well be - that the 2020 model redevelopment will stand the test of time.

A useful exercise for this hypothetical institution would be to run the 2019-era models in parallel with the 2020-era variants. If the older models start performing significantly better than the newer ones, it will be clear that the redevelopment exercise was premature and probably, on balance, unwise.

Moving to the other end of the spectrum, our example bank may want to fully understand the contours of the post-COVID world before an attempt is made to redevelop critical models.

Whenever there's an event that defines a generation - 9/11 certainly qualifies, and I suspect COVID-19 will as well - questions begin to be asked about how the world will fundamentally shift. I imagine that some elements of human interaction will change - work-from-home flexibility, for example - but most elements of post-pandemic life will be recognizable to pre-2020 time travelers.

If you want to accurately model the way COVID-19 has changed the world, you will have to wait until you have at least two or three years of post-pandemic data. The behavior that we witness as the pandemic chapter closes will one day be the subject of interesting and serious academic study - and the results of this will eventually filter into bank models.

Industry practitioners, however, do not have the luxury of simply waiting for more data to gather. Indeed, I suspect that regulators would be a little cross with a bank still using ill-fitting 2019-era models in, say, 2023.

Parting Thoughts

The optimal timing of model redevelopments will clearly be idiosyncratic, varying by country, company, product line and model type. In this article, we are simply making some general timing suggestions for a typical (average) model, whatever that mythical beast might look like.

With that said, if I were making these decisions and had some flexibility, I would like to know that the COVID-19 chapter was closed before kicking off a major model rethink. The timing of this may not be easy to nail down, though the apparent success of vaccine programs breeds increasing confidence of a COVID-19 finale sometime during the looming northern summer.

One suspects, therefore, that autumn will usher in a period of elevated activity in model building and validation teams around the world.

At that point, we won't have a complete understanding of likely post-COVID borrower behavior, but we will start to have suspicions. Heading into 2022, we should get a much clearer idea of whether we can place an asterisk next to the COVID era, at least where risk modeling issues are concerned.

Tony Hughes is an expert risk modeler for Grant Thornton in London, UK. His team specializes in model risk management, model build/validation and quantitative climate risk solutions. He has extensive experience as a senior risk professional in North America, Europe and Australia.