Model Risk Management at the Crossroads: Meeting New Demands with Limited Resources

Download

Summary

The number of models used by financial firms has risen significantly. This reflects many factors, such as regulatory requirements and the opportunities offered by advanced analytics to improve firms’ performance. Models are used to address a wide range of issues, from meeting regulatory requirements to identifying potential new customers and more.

Firms need strong model risk management functions. But in a world of limited resources, it is vital that model risk management resources are deployed efficiently, focusing on the material areas of risk. Although model risk management has advanced a great deal in the past 10 years, room for improvement exists, particularly if it is to keep up with the changing modeling landscape.

This document offers a few suggestions for practitioners and regulators and draws together information from the relatively few written sources to present an outline of the main elements of good model risk management. It presumes familiarity with the idea of model risk management and with principles of risk management.

Suggestions for practitioners

• Frequent monitoring of model fitness is needed, along with an organizational willingness to either abandon or rapidly revise the model when evidence of instability appears.¹ Monitoring is not the same as initial validation: It is focused almost entirely on fitness-for-purpose. ² Details of monitoring methods are likely to vary with the focus of the model. More focus on monitoring is needed, partly because of the increasing popularity of machine learning (ML) and similar advanced analytics. These can improve models, the model building process and model validation, but complexity increases as well, and ML methods have a greater chance of producing models that more rapidly become not fit for purpose. ML methods tend to overfit the data and thus may capture features of the data that are not robustly predictive of the phenomena of interest. Depending on the application, regular re-estimation may not be enough to control associated risks.

Suggestions for regulators

• Supervision and regulation of model risk management should not be mechanical, but should consider all of regulatory, business and risk management goals. Models should be prioritized for attention according to the severity of risks posed, and resources should be allocated accordingly. Once initial validation is complete, an effective ongoing monitoring process is needed to ensure models are fit for purpose. Occasional checks also are required to ensure that changes to models are properly reflected and policies and procedures are followed. Most written guidelines by regulators acknowledge that proportionality is appropriate for some elements of model risk management, but in practice the application of proportionality is too often limited by regulators.

• Regulators should also keep in mind that annual full revalidation of all models is neither efficient nor risk-sensitive. Few people have excellent skills in all the areas needed to do model risk management well. Annual full revalidation of all models spreads scarce resources too thin and leads to a program that is less effective. Similarly, initial validation and periodic revalidation of models that clearly pose little risk can be more streamlined.

The remainder of this document outlines the key elements of an effective model risk management program. The suggestions offered above are not repeated, but their substance is spread throughout.

Background

Model risk is the potential for adverse consequences from decisions based on the output of models or decisions made by models.

• A “model” is a system that converts assumptions and input data into quantitative estimates, decisions, or decision recommendations. Other definitions are possible.
• A firm’s definition is important because it determines what is subject to model risk management. The definition should be reasonable and should capture all sources of risk that can be cost-effectively managed by the methods described below.
• Cliff effects from the definition should be avoided. Risk management functions should recognize that systems outside the definition of “model” are not subject to model risk management but may pose risk and need the attention of risk managers. Businesses should recognize that the decision to use systems within the definition of “model” or outside that definition should include the quality of the associated machinery (including risk management), not just the quality of the system itself.
• The definition is similar to that in the 2011 U.S. supervisory guidance, but explicitly includes decision support tools that may not produce quantitative output. ³

The primary objective of model risk management is to ensure that models are fit for purpose, meaning the potential for adverse consequences is within acceptable limits.

• Models not fit for purpose should not be used, or should be used only in a limited, controlled way while alternatives are sought.
• Some elements of model risk management programs are also aimed at maintaining the firm’s capital stock of models (e.g., documentation requirements).

Organization and Governance

Business lines that use models should own the risks and be primarily responsible for ensuring that the risks are properly mitigated. However, because of potential conflicts of interest, businesses cannot be solely responsible for model risk management. Model risk management should involve other parts of the organization as well.

An effective governance structure is important to successful model risk management. Best practices include:

• Boards of directors and CEOs should visibly support model risk management functions. Tone from the top is important to effectiveness.⁴
• Organizations should have a written model risk management policy that sets out roles and responsibilities. It should include standards for the development, validation, monitoring and use of models.
• Model risk management activities should be audited.
• No perfect organizational location exists for the head of model risk management (chief model risk officer; CMRO). This is because all functions use models and thus any function may have conflicts of interest from time to time, if the CMRO is housed therein. It is most important that revenue-generating lines of business not house the CMRO because conflicts of interest are strongest there. As a practical matter, in most organizations the CMRO reports to the chief risk officer (CRO). One way to manage any remaining conflicts of interest would be for the organization’s risk framework to prescribe independence of operation for the CMRO and for auditors to assess such independence of operation and to report any weaknesses.
• Some activities that aid model risk management may be done by model owners, who often will be part of business functions.
• Heads of model validation teams should be independent of the functions that develop and use models and should report to the CMRO.
• Usage of models (and therefore users) should be controlled, meaning only uses are permitted for which a model has been deemed fit for purpose. New uses of an existing model should be evaluated for fitness for purpose. For uses clearly posing minimal risk, fitness for purpose can be determined with little analysis and perhaps mechanically.
• For models posing more than minimal risk, model developers, model users and model risk managers should jointly agree on a model’s fitness for purpose. Where they do not agree, an entity higher in the governance structure that is independent of the relevant businesses should decide.
• Reporting of model risks and model risk management activities is necessary for an effective governance structure and for effective risk management of the firm.

The activities, powers and responsibilities of model owners and developers, businesses and model risk managers differ:

TAXONOMY

• Businesses, model owners and model developers are typically in the first line of defense, while model risk managers are typically in the second line of defense.
• Different firms organize model risk management differently. The groupings in this document (model owners and developers, businesses, and model risk managers) and the assignment of activities to each group are indicative.

FIRST LINE…MODEL OWNERS AND DEVELOPERS

• Model owners and developers include those who implement models from vendors as well as those who build new models and modify existing models.
• Good knowledge of the business problem being addressed is essential.
• Models (and input data) should be documented by developers (and users) with sufficient clarity and detail that others can understand and operate the model. However, standards for documentation should make it cost-effective.
• Model owners may refuse to make changes or improvements suggested by model validation teams if such changes or improvements are admitted by model risk managers as being not material for a model’s fitness for purpose.
• Model owners and developers should update documentation as changes to a model are made, and should maintain records of changes.

FIRST LINE…BUSINESSES

• All models will one day no longer be fit for purpose because businesses, and the phenomena being modeled, change. Businesses should monitor each model for fitness on a schedule, using methods that are suited to the features of the model and the methods by which it was developed. Model risk managers should approve the monitoring plan. Models more likely to quickly or severely become not fit for purpose should be monitored more intensively. Users of a model should be trained to understand its strengths and weaknesses to aid monitoring.
• Each business should contribute to development and maintenance of an inventory of models.
• Conditional on a model’s being fit for purpose, businesses should make the tradeoff between model quality and costs of development and operation.
• Businesses should control access to and usage of models and their outputs to ensure that all uses are fit for purpose.

SECOND LINE (MODEL RISK MANAGERS)

• Model risk managers are those who validate models, who check that monitoring of model fitness for purpose is adequate, and who control the model risk management process.
• Model risk managers should check that businesses maintain adequate inventories of models. Contents of the inventory should support governance, validation and monitoring, and, for use in right-sizing monitoring and revalidation activity, ⁵ inventories should include a categorization of the degree of risk posed by each model.
• Model risk managers should validate each new model. Most models should be validated before entering use, but some, especially those posing low risk or for which usage is a prerequisite to validation, may be validated soon after use begins.
• The main purpose of validation is to evaluate fitness for purpose. Though model risk managers may suggest improvements, their job is not to cause the “best” model to be built.⁶
• Validation of new uses of an existing model should be done to ensure fitness for purpose but many other elements of full validation can be omitted.
  • A model should be revalidated: ⁷ (1) after material changes are made; (2) after new information calls into question fitness for purpose or the quality of relevant systems, such as usage controls; (3) on a regular schedule, presuming revalidation is not triggered earlier. The maximum time between revalidations should depend on the risk posed by the model. Annual revalidations are necessary only for high-risk models or where they are demanded by users or regulators. Stronger monitoring may support longer periods between revalidations.
• The team validating a model should include some people who, collectively, understand the business problem, the data, and the modeling methods at least as well as the model’s development team. This implies that model validation teams need some appropriately expert staff. The governance structure should evaluate the fitness of each team to do its job.

EFFECTIVE MODEL VALIDATION SHOULD INCLUDE

• Categorizing the amount of risk posed by the model and recording it in the inventory, for use in right-sizing monitoring and revalidation activity. Categorization should take into account the inherent risk posed by the model, the size of portfolios and income streams that are affected by such risk, and relationships with other models.
• Ensuring controls on usage are in place. The controls should be sufficient to trigger a limited revalidation if new uses occur.
• Ensuring change management controls and systems for recording changes are in place. Controls should trigger revalidation if sufficiently extensive changes are made.
• Examining the business’s plan for monitoring the model between revalidations for ability to detect loss of fitness for purpose.
• Identifying relationships among models. Examples of relationships include use of one model’s outputs as inputs to another model, or use of similar modeling methods across multiple models that pose similar risks. A model may be viewed as fit for purpose in isolation, but may nevertheless contribute unacceptably to the firm’s model risk due to relationships with other models. As an example, identical assumptions of uncorrelated housing prices in each of a suite of mortgage-related models, each addressing only one medium-sized sub-portfolio, might be acceptable for each model standing alone, but taken together might expose the firm to unacceptable model risk.
• Omitting or limiting in scope some validation activities, if a model poses low risk. Factors that could make a model high risk must be considered prior to any decisions on omissions or limitations.

Parting Thoughts

This document suggests a model risk management strategy in which resources are applied in proportion to risk posed. Some effort must be expended to determine that risks associated with a model and its uses are de minimus, but provisions should be made to stop tasks once such a determination is made and it becomes clear that the tasks are not cost-effective. As with all other risk management, the goal should be to manage model risk to an acceptable level, not to do everything possible to minimize the risk posed by every model.

Many operational aspects of model risk management, such as the manner in which a model’s risk is categorized or the details of monitoring and validation strategies, have not been mentioned in this document. These are important, but are likely to differ across firms and models.

¹ Some refer to “monitoring” as “review.”

² All models require monitoring because all models will one day be not fit for purpose. Monitoring focuses on things that can greatly affect model fitness for purpose, such as changes in a market being modeled. However, the intensity of needed monitoring differs across models. Some matters given attention during initial validation, including documentation, user controls, change management procedures, and other matters, generally require less frequent attention thereafter.

³ For example, a trading system that places orders autonomously might be viewed as producing buy and sell order rather than estimates. It should be subject to model risk management because it takes risk for the firm. An example of what is not included is accounting software, which applies rules rather than being based on assumptions.

⁴ Boards of directors should not be involved in the details of model risk management unless they choose a governance structure that sometimes requires it.

⁵ Categorizing models by level of risk can aid the allocation of model risk resources. Proceeding as if all models pose similar risk may not only increase costs of model risk management but also may degrade effectiveness of model risk management.

⁶“Conservatism,” such as increasing the severity of risk estimates reported by risk management models in an attempt to offset model weaknesses, is not a sufficient solution to the challenges of model risk management, nor does it remove the need for the many elements of model validation.

⁷ Secondary benefits of revalidation include detection of unauthorized uses of a model, ensuring that continued use of the model is supported by documentation, data management, and other procedures, and generating information to support evaluation of relationships among models.