"The early bird gets the worm, but the second mouse gets the cheese." In other words, those in the process of starting their current expected credit loss (CECL) adoption can learn from the earlier adopter's experience with CECL adoption, particularly during the model validation process.
The supervisory Guidance on Model Risk Management (SR 11-7/OCC 2011-12) was formally adopted by the Federal Deposit Insurance Corporation (FDIC) in June 2017 to specifically address smaller financial institutions. Many larger institutions have gone through the time-consuming, complex model validation process and learned what both validators and regulators are focused on when it comes to CECL models. The observations below are not a comprehensive list of requirements; however, the offer insights in preparation for the required model validation on the road to CECL adoption.
The Supervisory Guidance on Model Risk Management (SR 11-7/OCC 2011-12) was formally adopted by the Federal Deposit Insurance Corporation (FDIC) in June 2017 to specifically address smaller financial institutions. Many larger institutions have gone through the time-consuming, complex model validation process and learned what both validators and regulators are focused on when it comes to CECL models. The observations below are not a comprehensive list of requirements; however, they offer insights in preparation for the required model validation on the road to CECL adoption.
Model Theory Should Be Well Supported
Documenting the process to select the model that will be used to estimate the allowance for credit loss (ACL) is critical. It is also important to confirm that management can express why the selected methodology is the most appropriate choice for the bank’s portfolio. Finally, the Audit Committee should be well educated and informed as to the bank’s vendor selection and/or methodology, both for the quantitative model and the qualitative factors.
Qualitative factors and model overlays may be one of the most significant items for smaller banks to consider. We recommend avoiding some of the challenges that larger institutions encountered (i.e., putting qualitative factors last on the list) when developing and implementing the model. Determine the qualitative factor framework early in the model development process to allow sufficient time for incorporation into the overall ACL calculation. Under CECL, the quantitative model now includes a forward-looking element, such as economic forecasts that were previously considered qualitative adjustments under the incurred loss model. A thorough review should be performed on the methodology to ensure the factors in the qualitative model are not double-counting expected losses already considered in the quantitative model. A thoughtful approach to the qualitative framework should be performed and quantitatively supported wherever possible. Model overlays, or adjustments, should also be well supported.
Start Early With Data
CECL models may require years of data, and it takes time to determine if data quality is consistent across all periods. The bank should clearly identify and document all data input sources and develop procedures to confirm data accuracy on an ongoing basis. By doing so, the bank should be able to mitigate risks by pinpointing any data issues that may cause limitations or weaknesses in the model’s calculations or capabilities early in the model development process. If the data is limited, disorganized, has gaps or proxy data is necessary, it is even more important to prioritize the data gathering process early.
Understand All Model Calculations
When implementing a vendor model, the bank should collaborate with the vendor to understand the calculations and the outputs produced by the model. If the vendor has documentation of the calculations, leverage this information to enhance any internal model documentation.
It is important that all calculations that impact the comprehensive ACL calculation are documented. Even calculations that are performed outside of the vendor model (i.e., qualitative factor overlays, other “adjustments,” etc.) should be included in the internal model documentation.
Develop An Ongoing Monitoring Plan
Regular ongoing monitoring is critical when evaluating whether the model continues to perform as intended or may require calibration. Putting aside the economic impact of the pandemic, which no early adopter predicted, it is important to measure the model’s performance on a periodic basis through stress testing, benchmarking, actual versus predicted, etc. Early adopters were often overwhelmed with model development and implementation and therefore did not have time to consider the importance of an ongoing monitoring plan. Institutions are encouraged to develop and document an ongoing monitoring plan even if ongoing monitoring analysis has not yet been put into practice prior to implementation. Institutions should consider working with the model vendor to determine if they have elements or suggestions that can be leveraged when developing an ongoing monitoring plan. Below are considerations for components of an ongoing monitoring plan that could be implemented, as applicable to the model and institution:
- Test the completeness and accuracy of all data inputs.
- Evaluate changes in products, exposures and other factors.
- Create thresholds of acceptance that may warrant additional reviews and/or calibration.
- Compare estimates in credit losses to actual losses via back-testing.
- Benchmark using the vendor model’s functionality, if available, or perform the analysis internally (i.e., ratio analysis, benchmark to peers, etc.).
- Review quarter-over-quarter loss comparisons for probability of default (PD), loss given default (LGD), balance and total loss rate at pool level to identify ACL changes.
- Evaluate overlays and determine if they are masking information within the quantitative calculation that should be adjusted/recalibrated.
- Conduct stress testing for extreme conditions (e.g., pandemic shutdown). Refer to this article to learn more about post-pandemic CECL models. Stress test elements within the quantitative calculation, as well as the qualitative component.
- Perform sensitivity testing around assumptions and settings such as recovery lag, PD run out period, exposure at default (EAD), active state pool changes (if using a transition matrix), stability of loan risk grade, etc.
- Monitor model limitations identified during development to evaluate how those limitations impact or may impact the model overtime and if mitigating controls are still effective.
Perform Outcomes Analysis
Another factor that assists in the facilitation of a smooth adoption is an evaluation of the quantitative and qualitative trends driving the model results. While banks may be comfortable with the current results from the incurred loss model, previous adopters have learned that benchmarking the ACL results to the previous incurred loss model may not be ideal. The overall accounting methodology has changed, and it may not be appropriate for results to be similar between the two methods. Comparing how much of the ACL is derived from the quantitative portion, compared to the qualitative portion and how it varies quarter-to-quarter, is a good example of outcomes analysis. It may also be beneficial to take a step back and see how the changes in the ACL between periods compare to the movement in the overall economy.
Controls Are Key
Development of strong controls around assumptions, inputs and outputs is critical to the accuracy of the model results, especially when manual inputs or adjustments are required. Banks should confirm that reconciliation controls are adequately designed and effective at capturing different processing points for data throughout the modeling process. Periodic checks can help to improve data accuracy and completeness of the model. Read this article for more information.
When In Doubt, Document
As with any model, documentation is critical, and one of the most significant lessons learned by early adopters was how important complete and accurate documentation is to the entire model development, implementation and validation process. If you are unsure about the level of documentation for particular aspects of the model, think about the questions asked during development, along with questions anticipated from auditors, validators and examiners. Each development decision or piece of the puzzle that helps in the execution of the model should have documentation. The regulatory guidance states:
Without adequate documentation, model risk assessment and management will be ineffective. Documentation of model development and validation should be sufficiently detailed so that parties unfamiliar with a model understand how the model operates, its limitations and its key assumptions1.
Cecl Model Validation Documentation Recommendations
- Model background, including definition, purpose, use, theory, selection, merits and limitations
- Financial and operational impacts
- Key staff with roles and responsibilities surrounding model selection, development, testing, safekeeping, change authorizing and ongoing monitoring on the record
- Model data and security
- Model methodology
- Model risk rating and rationale
- Model limitations
- Model interdependencies
- Model assumptions
- Model inputs
- Model components, including processing
- Model outputs
- Model sensitivity
- Model quality, including targets and standards for acceptable levels of discrepancies
- Controls (change management, access controls, etc.)
- Model testing and ongoing performance monitoring
- Model use training and business continuity
How We Can Help
You have heard it before – don’t wait for 2022; start this process as early as possible. If your institution needs guidance regarding its model validation preparation and execution, DHG is a PCAOB-registered firm with a deep understanding of the requirements and compliance milestones needed for regulated banks like yours.
Use the following as a guide to plan your bank’s CECL model validation and implementation strategy.
- Identify the model and methodology selection(s)
- Develop a qualitative (Q) factor framework
- Data gathering
- Develop a training plan for model users
- Select a model validator
- Develop a model implementation timeframe with milestones
- Finalize assumptions and qualitative factors
- Draft model documentation
- Develop an ongoing monitoring plan and thresholds of acceptance
- Parallel run for results to be used in the validation including all Q factors and outside calculations
- Finalize the model documentation and facilitate approval by key stakeholders
- Validation performed using Q1 results
- Parallel run Q2
- Remediate findings and make adjustments/updates based on validation results
- Remediation testing by validators
- Parallel run Q3
- Perform ongoing monitoring
- Make any final adjustments
- Parallel run Q4
|Jan. 1, 2023 – Adoption of CECL Complete