“The early bird gets the worm, but the second mouse gets the cheese.” In other words, those in the process of finalizing their adoption of the CECL accounting standard can learn from the earlier adopter’s experiences with CECL adoption, particularly during the model validation process.
The Supervisory Guidance on Model Risk Management (SR 11-7/OCC 2011-12) was formally adopted by the FDIC in June 2017 to specifically address smaller financial institutions. Many larger institutions have gone through the time-consuming, complex model validation process and learned what both validators and regulators are focused on when it comes to CECL models. The observations below are not a comprehensive list of requirements; however, they offer insights to institutions in preparation for the required model validation on the road to CECL adoption.
Support Chosen Model Theory
Documenting the process to select the model that is being used to estimate the allowance for credit loss (ACL) is critical. It also is important to confirm that management can express why the selected methodology is the most appropriate choice for the bank’s portfolio. Finally, the audit committee should be well educated and informed as to the bank’s vendor selection and/or methodology, both for the quantitative model and the qualitative factors.
Qualitative factors and model overlays may be one of the most significant items for smaller banks to consider. We recommend avoiding some of the challenges that larger institutions encountered, i.e., putting qualitative factors last on the list when developing and implementing the model. Determining the qualitative factor framework early in the model development process can allow sufficient time for incorporation into the overall ACL calculation. Under CECL, the quantitative model now includes a forward-looking element, such as economic forecasts, that were previously considered qualitative adjustments under the incurred loss model. A thorough review should be performed on the methodology to help ensure the factors in the qualitative model are not double-counting expected losses already considered in the quantitative model. A thoughtful approach to the qualitative framework should be performed and quantitatively supported wherever possible. Model overlays, or adjustments, also should be well supported.
Identify Consistent Data Quality
CECL models may require years of data, and it takes time to determine if data quality is consistent across all periods. The bank should clearly identify and document all data input sources and develop procedures to confirm data accuracy on an ongoing basis. By doing so, the bank should be able to mitigate risks by pinpointing any data issues that may cause limitations or weaknesses in the model’s calculations or capabilities early in the model development process. If the data is limited, disorganized, has gaps, or proxy data is necessary, it is even more important to prioritize the data-gathering process and related controls early.
Understand All Model Calculations
When implementing a vendor model, the bank should collaborate with the vendor to understand the calculations and the outputs produced by the model. If the vendor has documentation of the calculations, the bank can leverage this information to enhance any internal model documentation.
It is important that all calculations impacting the comprehensive ACL calculation are documented. Even calculations that are performed outside of the vendor model, e.g., qualitative factor overlays, other “adjustments,” etc., should be included in the internal model documentation.
Develop an Ongoing Monitoring Plan
Regular ongoing monitoring is critical when evaluating whether the model continues to perform as intended or requires calibration. Putting aside the economic impact of the pandemic, which no early adopter predicted, it is important to measure the model’s performance periodically through stress testing, benchmarking, actual versus predicted, etc. Early adopters were often overwhelmed with model development and implementation and, therefore, did not have time to consider the importance of an ongoing monitoring plan. Institutions are encouraged to develop and document an ongoing monitoring plan even if ongoing monitoring analysis has not yet been put into practice before implementation. Institutions should consider working with the model vendor to determine if they have elements or suggestions that can be leveraged when developing an ongoing monitoring plan. Below are considerations for components of an ongoing monitoring plan that should be implemented as applicable to the model and institution:
- Test the completeness and accuracy of all data inputs.
- Evaluate changes in products, exposures, and other factors.
- Create thresholds of acceptance that may warrant additional reviews and/or calibration.
- Compare estimates in credit losses to actual losses via back-testing.
- Benchmark using the vendor model’s functionality, if available, or perform the analysis internally, i.e., ratio analysis, benchmark to peers, etc.
- Review quarter-over-quarter loss comparisons for probability of default (PD), loss given default (LGD), and/or balance and total loss rate at pool level to identify ACL changes.
- Evaluate overlays and determine if they are masking information within the quantitative calculation that should be adjusted/recalibrated.
- Conduct stress testing for extreme conditions, e.g., pandemic shutdown. Stress test elements within the quantitative calculation, as well as the qualitative component.
- Perform sensitivity testing around assumptions and settings such as recovery lag, PD run-out period, exposure at default (EAD), active state pool changes (if using a transition matrix), stability of loan risk grade, etc.
- Monitor model limitations identified during development to evaluate how those limitations impact or may impact the model over time and if mitigating controls are still effective.
Perform Outcomes Analysis
Another factor that assists in the facilitation of a smooth CECL adoption is an evaluation of the quantitative and qualitative trends driving the model results. While banks may be comfortable with the current results from the incurred loss model, previous adopters have learned that benchmarking the ACL results to the previous incurred loss model may not be ideal. The overall accounting methodology has changed, and it may not be appropriate for results to be similar between the two methods. Comparing how much of the ACL is derived from the quantitative portion, compared to the qualitative portion and how it varies quarter to quarter, is a good example of outcomes analysis. It also may be beneficial to take a step back and analyze how the changes in the ACL between periods compare to the movement in the overall economy.
Check Controls
Development and documentation of strong controls around assumptions, inputs, and outputs are critical to the accuracy of the model results, especially when manual inputs or adjustments are required. Banks should confirm that reconciliation controls are adequately designed and effective at capturing different processing points for data throughout the modeling process. Periodic checks can help to improve data accuracy and completeness of the model.
Document the Whole Process
As with any model, documentation is critical. One of the most significant lessons learned by early adopters was how important complete and accurate documentation is to the entire model development, implementation, and validation process. If you are unsure about the level of documentation for particular aspects of the model, think about the questions asked during development, along with questions anticipated from auditors, validators, and examiners. Each development decision or piece of the puzzle that helps in the execution of the model should have documentation. The regulatory guidance states:
Without adequate documentation, model risk assessment and management will be ineffective. Documentation of model development and validation should be sufficiently detailed so that parties unfamiliar with a model understand how the model operates, its limitations, and its key assumptions.1
CECL Model Validation Documentation Recommendations
- Model background, including definition, purpose, use, theory, selection, merits, and limitations
- Financial and operational impacts
- Key staff with roles and responsibilities surrounding model selection, development, testing, safekeeping, change authorizing, and ongoing monitoring on the record
- Model data and security
- Model methodology
- Model risk rating and rationale
- Model limitations
- Model interdependencies
- Model assumptions
- Model inputs
- Model components, including processing
- Model outputs
- Model sensitivity
- Model quality, including targets and standards for acceptable levels of discrepancies
- Controls (change management, access controls, etc.)
- Model testing and ongoing performance monitoring
- Model use training and business continuity
How FORVIS Can Help
You may have heard it before, but time is short. If your institution needs guidance regarding its model validation preparation and execution, FORVIS is a PCAOB-registered firm with a deep understanding of the requirements and compliance milestones needed for regulated banks like yours.
FORVIS looks forward to helping your institution with the implementation of CECL before the January 2023 adoption.
Reach out to us with any questions or to discuss your CECL implementation progress via the Contact Us form below. You can also visit our Financial Services page for additional resources.