Controlling Model Risk
By Margaret Elliott
For the top managers at NatWest Capital Markets and Bank of Tokyo/Mitsubishi, it was egg-on-the-face time. In March, Bank of Tokyo/Mitsubishi announced
that its New York-based derivatives unit was taking an $83 million after-tax
write-off because a computer model overvalued a portfolio of swaps and options
on U.S. interest rates. A few weeks later, NatWest Capital Markets announced
that it was taking a £50 million hit because of a mispriced portfolio
of German and U.K. interest rate options and swaptions run by a single derivatives
trader in London. After further investigation, the hole grew to £90
million. And then all at once, rumors about multimillion dollar losses related
to derivatives book mispricings at UBS, BZW and elsewhere were flying around
Don't jeer, and try not to point fingers. Given the range of models,
assumptions and data used in every bank and every dealer each day around
the world, it's likely that most firms are bound to have an "unfortunate
profit/loss event" lurking somewhere in their books.
Everyone from risk management consultants to newspapers calls this problem "model risk," an imprecise term that belies the multifaceted nature
of the problem or problems. But model risk covers at least three distinct
areas: the choice, testing and safekeeping of the mathematics and computer
code that form a model; the choice of inputs and calibration of the model;
and the management issues associated with these activities.
Each of these presents a huge challenge, and few observers can point
to financial institutions that have all these areas under control. "Maybe
one or two institutions that have been in the derivatives business from
the beginning are up to speed on the model risk issue," says one former
head of a swaps group. "But certainly no banks that have gotten in
the business recently." "Very few do it well," concurs Michael
Haubenstock, a partner at Price Waterhouse.
Ordering the code
It isn't a surprise that the use of computer models should pose a health risk to banks. The derivatives business would be unthinkable without the
elaborate models used to price its enormously complicated instruments. It's
easy to forget that the whole system is built on the intellectual labor-and
assumptions-of a few key people. "The actual analytics in a model come
from the quants on the trading desk, the front office," says Zahid
Ullah, head of derivatives investment management at Gotham Derivatives Corp.
and a former NatWest risk manager. "It could be a new version of an
old model, for a similar product, or a new spreadsheet model. They give
this to the back office for pricing-with no checks. The traders run the
show, and if they're challenged, they simply say it is proprietary."
Companies that use a model developed by a trader to value positions for
both the front and back office are asking for a rerun of the Nick Leeson
problem-a trader assigned to watch himself. The solution? The back office
develops or buys its own models to serve as an independent check on the
front office, usually with the aid of a high-priced independent risk management
group or outside consulting firm. Although that sounds easy, developing
a system to manage and test old and new models involves a number of difficult
The first step is pure common sense: to understand how models affect
your business, you must keep careful track of all the models you use. That
means keeping records of which models are used, who uses them and how they're
used. It also means keeping track of who keeps the code and who is allowed
to change the code.
Once that inventory is complete, it's important to control access to
those models. "We suggest locking up approved versions of the models
to facilitate control," says David Lukach, a partner in the capital
markets practice at Coopers & Lybrand. "Some of the problems we
see involve derivations of models-someone changes the logic in the model
and an inappropriate result occurs. Or a new product is developed and an
older model is used to price it with or without authorized modifications."
Models, of course, shouldn't be kept under lock and key. They should
evolve and improve with time. That means establishing a procedure for changing
the code. "Markets change and models need to reflect these changes,"
says Jitendra Sharma, director of derivatives and risk management at Arthur
Andersen in New York. "Making sure that each model change is verified
and implemented across all appropriate models keeps the bank up-to-date."
One important step-perhaps the most important-is developing a procedure
to test each of the models in use, preferably before they're used.
This is a complicated procedure, but one that pays off in spades. For
simpler, more liquid instruments, the model produces a price that can be
immediately checked against the market. Indeed many traders don't even bother
to use models for many simple options.
But as the instruments become more complicated, longer in term or less
liquid, traders may develop pricing models that reflect their own thoughts
about how these products will trade. The back office, however, may use different
models, either older or, in some cases, one that is more in line with market
standards. "For more liquid instruments, there may be a market standard
model-say Black-Scholes for options-that is routinely used for pricing,"
says PW's Haubenstock. "But while the back office uses that to price
the book, the traders may use a different model, one that makes adjustments
for the shortcomings of Black-Scholes. The two should use the same, approved
model, or else there will be an internal control problem and a potential
for unexplained profit-and-loss differences between front and back office."
For more complicated instruments, the starting point is the code and
the math. Institutions increasingly are turning to the Big Six or to independent
consultants for risk management checks that include model audits. And not
surprisingly, demand has soared since the NatWest announcement.
Outsiders who come in often try first to rebuild the model from scratch. If that is successful, they then run a series of bench-marking tests that
measure the dealer model against their own market models. "We see a
broader range of market models than a single bank, so our models may be
more up-to-date in terms of quant theory," says Andersen's Sharma.
"This can tell us whether the model the bank proposes to use is simply
wrong-that is, if it does not reflect or replicate the behavior of the underlying
instrument and therefore is inappropriate to the product."
Dealers that assign their internal risk management groups to vet their
models often purchase externally-developed software to check on their internally-developed
models. But if derivatives software doesn't offer the ability to bench-test
using the same inputs, it runs the risk of being just like the trader's
proprietary spreadsheet-unverifiable. "It is important that system
vendors give customers the option of dynamically inserting new models into
those systems," says Mark Garman of Financial Engineering Associates,
a top derivatives software house. "This allows institutions the opportunity
of using and comparing three different sources of models, namely the vendor
models, the customer models and the third-party vendor models." Some
vendor models do not offer this flexibility.
"A few years ago we saw mostly external auditors interested in purchasing our models as a check on the front office," says Emmanuel Mond, managing
director at Monis Software. "In last few months, we've had more purchases
by internal auditors who perform spot-check audits rather than trying to
running every trade though the model."
When auditors find disparities, they go back and check the numbers more
carefully. "We're not seeing very large differences, which indicates
that models are standardizing to some extent," notes Mond. "But
you've got to compare oranges with oranges. If you're pricing off a Black-Scholes
and they're pricing off of a Heath-Jarrow-Morton, you're bound to have disparities."
In the inevitable push mepull you that occurs in the process of
vetting models, one issue stands out-simplicity. "If you make the model
overcomplex, you embed risk that you often cannot easily verify," says
Simon Moss, head of the trading and risk management practice for IBM North
America. "If I assume constant volatility, for example, that says to
me that I am modeling a term structure. For instruments with optionality,
the common assumptions should be stochastic in nature."
In designing and vetting models, it's important to remember that models
should serve as guides for fast-thinking traders in dynamic markets. No
model ever completely replicates the dynamic changes in an instrument. In
modeling a basket option on the OEX, for example, IBM's Moss says, "When
seeking volatility for the basket, it would be inappropriate to supply the
volatility for each element." Andersen's Sharma puts it this way: "In
the model-development process there is necessarily a trade-off between replicating
the market exactly and over modeling. It's important to find a simple and
elegant way of accurately reflecting the market with the minimum number
of inputs and iterations." Don't overcomplicate unnecessarily seems
to be a rule of thumb.
Most problems with models do not occur in the code, but in the data,
inputs or calibration of the model in use. Do not underestimate the complexity
of the problem. For longer-dated illiquid instruments, it may be quite some
time before an input problem can be identified. And Value-at-Risk calculations
won't pick it up.
Take NatWest's much-publicized £90 million ($138 million) loss.
One popular theory is that the bank was using a single volatility for all
strike prices in its sterling/Deutsche mark over-the-counter swaptions book.
If the bank had used different volatilities for each strike, a so-called
volatility smile, the book would have been priced more closely with the
market and NatWest wouldn't have incurred the mispricing loss. "Particularly
for out-of-the-money deals, the smile effect gives a higher implied volatility,"
says Leon Borodovsky, Ph.D., executive director of the Global Association
of Risk Professionals and a risk manager at Credit Suisse First Boston in
The magnitude of the loss may have been a reflection of the length of
time it took the bank to pick up the inappropriate calibration. Observers
suggest that the incorrect volatility may have been used on the book since
late 1994, or about when the bank entered this particular business. "Though
I have no independent knowledge of the NatWest situation, an incorrectly
calibrated model will have far less effect when the book of business held
by the institution is either completely long or completely short,"
says Anthony Neuberger, a professor at London Business School. "In
my experience, the problems occur when a bank or dealer makes the business
decision to make a market in a particular instrument, going both long and
The other factor that shines out of the NatWest affair is the seeming
lack of consistency in the inputs used by the bank. "If flat volatility
was used in the sterling/Deutsche mark book, why wasn't it used in the sterling/dollar
book or the Deutsche mark/dollar book? What did those traders know that
that the sterling/Deutsche mark traders didn't?" asks Gotham Derivatives'
The best way to check model calibration is through stress testing. Running simulations with data that reflect different market scenarios should help
determine what inputs are appropriate for which book of business.
But like code testing, it is imperative to use more than one model while stress testing. "By using an independent model or a market-standard
model, you are essentially benchmarking the results for the same inputs
to the models and evaluating the potential differences in valuations,"
says Dilip Kumar, senior manager at KPMG. "Of course, you expect that
the results are quite close. But the interesting part occurs when there
are differences. To the extent that there are appreciable differences, you
are looking to interrogate the differences and fully understand the sources
of the differences so that you can form an opinion of the model being tested.
This is what educates you as to the strengths and weaknesses of the models,
so that you are aware of the sensitivity of the valuation to each of the
inputs. This becomes an important exercise when you are dealing with illiquid
transactions and not easily verifiable inputs."
Stress testing often forces certain peculiarities of models into high
relief-making them appear like calibration errors. "If a structured
derivatives desk is using a lattice pricing model, certain calibration values
may imply a relatively low hedging rate. That is because that value corresponds
to a point on the lattice itself. But another quite close value that falls
between the lattice points may imply a huge hedge," says Dave Penny,
vice president of software development at Algorithmics. "If you interrogate
the results you'll see that the huge implied hedge occurs in between the
lattice points. In all likelihood, it can be ignored." This concept
is very important to Tanya Styblo Beder, principal of Capital Market Risk
Advisors Inc. "We look for drivers of difference-it's an early warning
system that tells you when a model might be misleading," she says.
Where's the data?
It may be common sense, but consultants say few organizations make sure
that all their traders work off the same data. "We suggest that the
firm set up a centralized data capture mechanism," suggests Algorithmics'
Penny. "At the end of the day, for instance, external data come into
one place and then are fed into the necessary models to allow for an independent
double-check on the data the traders were using."
At CIBC in Toronto, explains Michael Crouhy, vice president for global
analytics in the market risk management group, implementing a first-class
financial rates database was one of the bank's key decisions in entering
risk management. "We use our own database to calibrate pricing models
and to feed our VAR system. We spent considerable of effort ensuring the
integrity and accuracy of our data. Using poor data can ruin the benefit
of sophisticated analytical models-garbage in, garbage out. We also insist
on full consistency between our pricing models in the front and middle offices,
and for profit-or-loss calculation, as well as in the VAR system. When we
develop risk management calculators at the desk level, the risk assessment
is consistent with what is produced by the VAR model. This greatly contributes
to the credibility and the acceptance by the traders of our VAR system,
especially when it comes to the calculation of capital charges. It was easier
for us to implement this approach as we started with a blank page two years
Few banks have that luxury and may be using not only a wide variety of
legacy systems, but also many data sources and incompatible historical information.
It's almost axiomatic that banks bring in data from manysources, and that
testing is done using historical data that may not match the detail of data
feeds available today. "It is imperative to stay up-to-date with best
practice as far as data and quantitative methods are concerned," says
Beder. "That means determining which data, which sources and which
models are used. We've only been technologically advanced enough to short-circuit
keypunch error for a few years; often older data are full of errors or no
longer relevant to the new realities of the marketplace."
At Bank of Tokyo/Mitsubishi, at least some of the losses resulted from
models and procedures that weren't kept up to date. "Take the swaption
market as an early example," says Beder. "When they started, swaptions
were too small to have their own volatility, so capital markets volatilities
were used instead. Eventually swaption volatilities evolved. If your model
was still using capital markets volatility when the market moved on, then
it would give you the wrong answer."
Many model risk issues become amplified when dealers extend existing
businesses or enter new ones. Dealers that trade or make a market in a product
for the first time should make a special effort to reassess models, procedures,
data and best practice before they jump in. In fact, any significant change
in the way you do business should trigger a "rethink." "In
our experience, it is valuable to reverify models on a periodic basis given
the dynamic nature of the derivatives business, and advisable to verify
model imputs by comparison with independent sources, on a frequent basis
(daily, weekly or monthly)," says Coopers & Lybrand partner Mark
Casella. But that may just be routine housekeeping.
A change in the direction of the model data should encourage a more in-depth investigation. After all, models that are required to support a market-making
operation are several degrees more complex than those required for a simple
presence in the market. Understanding how to make that leap requires conceptual
as well as technological support.
Here's where external consultants can help. CMRA's Beder outlines the
process she puts new models in new businesses through. "First we compare
the math engine and results of the client's new model with models we build
at CMRA. Once we've determined that the model is robust, we further evaluate
it to answer a variety of questions," she explains. These questions
include what variables, given a small move, cause a large move in price
or risk valuation of the position or portfolio? What variables or exposures
are considered to offset each other? By how much? What is this model's acceptance
in the marketplace? Does the majority of the market use the same data inputs
and modeling assumptions (for example, raw data, curve building, interpolation
or extrapolation techniques, bid/ask levels, volatility assumptions, mapping
or other simplifications)?
Beder suggests that these questions should be answered on a periodic
basis for all models. "For example, a classic problem that arises is
the use of at-the-money volatilities to perform pricing on options portfolios
that may have moved to in-the-money or out-of-the-money status (or will
move to this status under stress-testing assumptions). Answering the questions
above often prevents unexpected model losses by revealing the degree to
which the use of at-the-money volatilities-or a lack of a term structure
to volatilities-matters." Beder says that to the degree that it is
significant in the valuations or risk views created by the models, changes
can be made. To the degree that the error is insignificant or within a palatable
range, the firm may be able to avoid expensive technological or operational
The establishment of independent risk management groups (which report
directly to the board) to evaluate models, rather than doing so through
the trading profit centers, has been a widely applauded effort. But internal
risk management groups will always trail traders. "The expertise resides
on the trading floor," says KPMG's Kumar. "Few firms have equally
talented people on the control side. Risk management needs to stand apart
from the traders. Models must be validated independently."
The answer, it seems, involves strong checks and balances. "We work with the traders, but we have the right to challenge them as well,"
says CIBC's Crouhy, who heads this effort at the Canadian bank. Just how
much of a right depends on the power invested by the board in this effort.
CIBC uses external consultants as well. "Risk management comes from
the top of organizations," says Ken Young, vice president of Technological
Solutions Co. "There needs to be an incentive for middle management
to make a profit and control the losses, not just to make a profit."
Not all banks have been successful in empowering their middle offices,
and thus rely more heavily on the external consultants. What this implies,
according to one consultant, "is that senior management doesn't know
who to listen to within their own organizations."
Intensive model auditing, stress testing and smart risk managers are
all necessary-but they aren't enough. All the math geniuses in the world
don't help if management either neglects to implement the procedures necessary
to produce accurate calculations of risk or ignores those outputs. In all
the recent derivatives losses, management can be faulted for a lack of understanding
of the problem.
An earlier conceptual step is necessary: understanding the nature of
the businesses in which the risk is incurred. Observers of the recent travails
at NatWest Markets and Bank of Tokyo/Mitsubishi suggest that while the actual
culprit in these losses is indeed a form of model risk, the management at
these institutions-and many others besides-may not have understood the full
range of risks associated with their businesses. "Every time you make
a mathematical assumption, input data from an external feed or estimate
the value of an illiquid private placement, you are incurring potential
model risk," says CMRA's Beder. "That said, three-quarters of
the situations we've seen recently that led to derivatives losses were attributable
to model risk." Beder estimates that $1.9 billion in recent publicly
declared derivatives losses by financial institutions are model-related.
Beder makes this point to emphasize that not all model risk problems
are associated with the longer-dated, highly complex and illiquid instruments
that were present in the NatWest and Mitsubishi cases, or the earlier revelations
of smaller losses by BZW and Nationsbank on currency trades. Although the
risk of mispricing may be greatest on these products, they typically form
less than 10 percent of the derivatives book. Problems with data or assumptions
could have a much greater impact if they appear in association with a more
mainstream form of derivatives business.
Model risk can't be eliminated entirely. It's a problem that mutates
as markets change. What's required is a true understanding of the breadth
and depth of the problem-it affects every nook and cranny of an institution's
derivatives business. Eternal vigilance is necessary as well. Beyond the
mechanical procedures of testing models and calibrations and vetting the
accuracy of data inputs, banks need to find a happy tension between the
honest and intellectually curious risk manager and the profit-seeking trader.
|The Cost Of Fudging It
As the NatWest and Bank of Tokyo Mitsubishi losses make clear, it's critically important to check and update the input variables going into a model. Before
pricing a portfolio, for example, it's usually necessary to recalculate
the zero curve, and enter the most current foreign exchange rates, as well
as the prices of all the underlying instruments. Sometimes, when all the
data are not available for the type of instrument priced, a trader may decide
to input data relevant to a similar, but not exactly the same, instrument.
Here's what would happen if you used the volatility of the three-month
forward rate to price a semiannual resetting cap. Table 1 shows the resulting
errors based on a six-month caplet starting six months from today. The notional
amount is $1,000,000, the strike rate is 6.6 percent (making the cap slightly
in-the-money) and the volatility is assumed as 10 percent. If instead, a
volatility of 12 percent is input into the valuation model (a 20 percent
error, which is not that uncommon), the error in the estimation of the caplet's
price is 21.23 percent.
The impact of the input error would be amplified if the option was
far out-of-the-money. This is shown in Table 2, where everything is kept
the same as before, except the strike rate is now 7.5 percent. Under this
scenario a 20 percent error in the input (volatility) causes a 155 percent
error in the valuation. Taking into account that we are only considering
one caplet in this example and that portfolios usually comprise hundreds
or thousands of caps, swaps, bonds and other instruments, one can readily
see that the overall error could be substantial.
-Frank Fronzo, INSSINC Software
Table 1. Errors in the caplet's premium when the
option is slightly in-the-money.
Table 2. Errors in the caplet's premium when the option is far out-of-the-money.