The Evolution of Model Risk Management

The Evolution of Model Risk Management: From Regulation to Practice 

December 18 | By Josh Goldberg

Simon alumnus Josh Goldberg explains the 25-year journey of model risk management. 

Regulators of financial institutions play a vital role in encouraging prudent risk management. Yet designing an effective regulatory framework that balances innovation with accountability, while also establishing appropriate guardrails, is no easy task. Regulations can sometimes miss the mark, creating layers of bureaucracy rather than fundamentally changing the way risk is managed.
Model risk management has emerged as a compelling example of effective, common-sense regulation. The best practices that evolved through bank regulation are now broadly applicable across industries. In this post, I’ll explore the 25-year journey of model risk management’s development, which has closely paralleled my own career.

Early Foundations: The Establishment of Model Validation

In 2000, the Office of the Comptroller of the Currency (OCC) introduced model validation as a foundational practice for managing model risk. At that time, independent model validation was the only established method for model risk management. OCC Bulletin 2000-16 introduced the concept of “effective challenge,” which requires model validators to actively question a model’s explicit and implicit assumptions. Much like the audit principle of professional skepticism, it encouraged a deeper review process, systematically decomposing models for rigorous testing.

This model validation standard was in place when the 2008 Great Recession struck. The crisis led many to question why models had not accurately predicted such significant losses. During this time, I was working in credit model risk management, gaining firsthand insight into potential shortcomings. Although models indeed required more robust challenge, the deeper issue was a cultural one. Risk culture—especially the interaction between executive management and model results—often affected outcomes more profoundly than technical model shortcomings. Governance structures and cultural attitudes in the C-suite, as well as pressures to present palatable results to investors, made genuine risk transparency difficult to achieve.

Expanding the Scope: The Rise of Internal Audit

In 2009, Freddie Mac hired me to develop an approach that expanded beyond standard model validation,  “black box” approaches  which only tested inputs and outputs without probing model assumptions. Instead, Internal Audit was tasked with directly evaluating the model validation function and assessing the reasonableness of model assumptions and results. This additional level of independent review helped safeguard against biases resulting from the close relationships between model developers and validators.

Growth in Enterprise Risk Management (ERM)

After the Great Recession, Enterprise Risk Management (ERM) departments expanded to cover a wider array of risks, from credit to market and liquidity. At organizations like Fannie Mae and Freddie Mac, this growth led to more oversight roles in Legal, Internal Audit, and ERM, while trading floor activity waned. However, with this expansion came challenges, as ERM departments often became too broad and unfocused, resulting in redundancy and loss of agility.

A More Integrated Approach: The Three Lines of Defense

The release of the Supervisory Guidance on Model Risk Management, OCC 2011-12 (also known as SR 11-7), marked a significant turning point. This interagency guidance firmly established that model risk should be managed like other types of risk. Freddie Mac had already laid the groundwork, aligning its model risk management practices with what would later be recognized as industry best practice.

In 2014, the OCC’s Heightened Standards further formalized the “three lines of defense” model, shifting more risk management responsibilities to the first line, helping ensure that those closest to business operations were actively managing risk rather than relying on second-line oversight alone. This shift required an adjustment period as business lines, previously accustomed to outsourcing risk management, adapted to new expectations.

Emerging Risks in a New Era of Model Risk

As the industry adopts modern modeling techniques, new risks have emerged, prompting updated regulatory guidance. In 2022, the Federal Housing Finance Agency (FHFA) issued an Advisory Bulletin on Artificial Intelligence and Machine Learning Risk Management. Reflecting an ongoing partnership with industry stakeholders, this guidance anticipates the unique governance requirements that AI/ML models bring, particularly around explainability and ethical use.

I represented Summit Consulting on a panel  at the 2024 FHFA Model Risk Conference, discussing explainability, transition to new credit scores, and handling historical data from the pandemic. My focus was on the unique challenges explainability brings to model risk management, especially where regulatory requirements intersect with practical business needs. For certain applications, such as financial reporting or fair lending, explainability is non-negotiable. In other cases, like fraud detection, flexibility to adapt to new patterns may require less transparent methods. Managing explainability involves making complex methodologies more intuitive to users and stakeholders, balancing performance and interpretability as context demands.

Dan Keating, Assistant Professor and Faculty Director of Academic Support at Simon Business School, presented on how Generative Artificial Intelligence would transform the education of future business leaders.  His insightful talk is part of a broader Generative AI initiative by Simon Dean Sevin Yeltekin.  The breadth of the speakers from technology providers to educators demonstrates how impactful AI will be to the practice of model risk management.

Looking Ahead

Over the past 25 years, model risk management regulations have evolved considerably, enhancing transparency and oversight. As financial institutions adopt increasingly complex models, effective regulatory engagement with industry will be critical to addressing the challenges ahead. Bank regulators face the ongoing task of staying current with these rapid advancements in methodology, but they have demonstrated a strong track record thus far.

Simon alumnus Josh Goldberg serves as Head of Model Development at Summit Consulting, LLC. In this role, he leads the firm’s data science practice, applies industry best practices to deliver value to Summit’s clients, and mentors staff to foster professional growth.
 

Follow the Dean’s Corner blog for more expert commentary on timely topics in business, economics, policy, and management education. To view other blogs in this series, visit the Dean's Corner Main Page.