.
.--.
Print this
:.--:
-
|select-------
-------------
-
Comparing Add-in Accuracy

Our intrepid modeler puts eight packages through their paces—and finds some significant pricing differences.

By William Margrabe

My 10-day odyssey through an often mysterious and sometimes frustrating world of computer hardware, Windows 95, Excel and packages of add-ins is over. While we wait for a modern-day Homer to chronicle my exploits in verse, this article summarizes how I navigated the perils on that trip, and how you might want to deal with the same issues.

My original plan was to design a diverse portfolio of equity, currency, commodity and fixed-income derivative products; load all the add-ins simultaneously; price each product with the relevant function from each package; and compare the results. That plan was too ambitious—like planning to climb Mount Everest, solo, one day after lunch, without oxygen.

In the time I had allotted for this project, I managed to price 10 deals with up to eight of the packages and my proprietary software. I will report in detail on only three relatively simple trades: a European put option, an American put option and a European put on a spread with a nonzero strike. For reasons that I hope will become obvious, fixed-income derivatives must wait for another day.

Coaxing the add-ins to give the same value for a price or Greek wasn’t nearly as easy as I had anticipated—even when they would eventually agree to several decimal places. I kept on investigating, however, until all the packages sang the same tune or until I knew why they didn’t.

The differences came from three main sources: inputs, algorithms and outputs.

The eight packages interpreted a given input rate of interest or dividend yield three different ways. One package wouldn’t accept dividend yield as input.

The eight packages had at least three different algorithms for converting input dates into years to expiration. While all eight packages appeared to use the Black-Scholes-Merton model for pricing European puts, they differed visibly on some of the Greeks. They used four algorithms for pricing American puts, and eight algorithms to price the spread option. Sometimes the algorithms weren’t what the documentation said. Sometimes the documentation didn’t say.

There were also some significant differences in the methods of reporting sensitivities to interest rates. Some vendors reported more sensitivities—sometimes many more—than others. Some provided sensitivity to a move in an annual percentage rate, while others assumed a continuously compounded rate was moving. Sometimes the sensitivities weren’t as documented.

The European put

We would all expect a fair amount of agreement on the pricing of European options for three reasons: European options have been around for a long time; the standard “closed-form” model for pricing them has been available for more than 25 years; and numerical algorithms that the Black-Scholes-Merton model requires have been around for decades.

My initial results for premium and Greeks appear in Table 1. The different packages—anonymous at this point—did not all produce the same values. Five packages agree that the premium should be 5.5913. Two agree on 5.6049. One says 5.5886. That raised questions: How can we quantify the disagreement? Why don’t all the add-ins agree to four significant figures on the premium and all the Greeks? Do “right answers” exist? If so, what are they? How much disagreement should we tolerate? Answering all these questions completely took several days.

Standard deviation—slightly more than one part in a thousand for premium—is a way to measure variability of results across vendors. The variability of sensitivity to interest rate and dividend yield is much larger. The apparent frequent lack of a number describing sensitivity to a change in dividend yield is startling.

To answer the remaining questions, I studied the documentation—and spoke with experts, where required—for clues about models, numerical algorithms and definitions of inputs and outputs.

Inputs

Table 1 European Put Option Value and Greeks
Model Premium Delta Gamma Kappa Rho_Div Rho_Int Theta
FEA 5.5913 -0.4523 0.0255 51.0503 90.4540 -101.6367 -0.9967
FinCAD 5.5913 -0.4523 0.0255 51.0500 -96.6840 -0.9967
Intermark 5.5913 -0.4523 0.0255 51.0500 -101.6404 -0.9967
Marvin 5.5886 -0.4577 0.0255 50.8849 -102.8093 -1.0468
MBRM 5.5913 -0.4523 0.0255 51.0500 -101.6404 -0.9967
Monis 5.6049 -0.4534 0.0256 51.1737 86.3596 -97.0355 -1.0059
Savvysoft 5.6049 -0.4534 0.0256 51.1737 86.3596 -97.0355 -1.0059
Tech Hackers 5.5913 -0.4523 0.0255 51.0500 90.4584 -101.6410 -0.9967
Mean 5.5944 -0.4532 0.0255 51.0603 88.4079 -100.0154 -1.0053
Std.Dev. 0.0065 0.0019 0.0000 0.0902 2.3652 2.5972 0.0173

Different vendors define and handle inputs differently. Sometimes, this is inconsequential. For example, some systems refer to an option’s “maturity,” rather than its “expiration.” Some systems accept time to expiration in years, others accept a value date and an expiration date, and some want or allow other inputs.More significant were differences in handling interest rate and dividend yield. The FEA, Intermark, Marvin and MBRM systems take their inputs the way I happen to be used to inputting them—as continuously compounded annual rates (such as 0.05). FinancialCAD and Savvysoft, however, want to receive dividend yield and interest rate as annual percentage rates (such as 0.05127 = exp(0.05)—1). The Monis system requires APRs to be expressed as percents (such as 5.127, rather than 0.05127). The Tech Hackers system asks for a continuously compounded interest rate, but expects a continuously compounded “holding cost”—the cost of carry—that equals the interest rate minus the dividend yield. After correcting for my misunderstanding about conventions for dividend yield and interest rate, the premium values for all but one package agreed to four decimal places. Two days later, I learned how to input time properly for the remaining package, and all premium values agreed to four decimal places, as in Table 2.

Table 2 European Put Option Value and Greeks—Revised
Model Premium Delta Gamma Kappa Rho_Div Rho_Int Theta
FEA 5.5913 -0.4523 0.0255 51.0503 90.4540 -101.6367 -0.9967
FinCAD 5.5913 -0.4523 0.0255 51.0500 91.3630 -101.6411 -0.9967
Intermark 5.5913 -0.4523 0.0255 51.0500 -101.6404 -0.9967
Marvin 5.5913 -0.4523 0.0255 51.0503 -101.6411 -0.9967
MBRM 5.5913 -0.4523 0.0255 51.0500 90.4513 -101.6404 -0.9967
Monis 5.5913 -0.4523 0.0255 51.0500 90.4584 -101.6411 -0.9967
Savvysoft 5.5913 -0.4523 0.0255 51.0500 90.4584 -101.6410 -0.9967
Tech Hackers 5.5913 -0.4523 0.0255 51.0500 90.4584 -101.6410 -0.9967
Mean 5.5913 -0.4523 0.0255 51.0501 90.6370 -101.6402 -0.9967
Std.Dev. 0.0000 0.0000 0.0000 0.0002 0.4058 0.0016 0.0000

Different algorithms

Based on the apparent inputs, I guessed that each vendor used a variation on the Black-Scholes-Merton model, and probably the same algorithm. The identity of that algorithm wasn’t obvious, however. To discover it, I computed premium and Greeks with eight of the algorithms that I use when reviewing models for clients. The premium and Greek results for my “closed-form” or “analytic” Black-Scholes-Merton algorithm agreed to four decimal places with those of all packages, so I conclude that they all use that algorithm, or a similar one.

Table 3 A Translation Table of Names For Greeks
Input Underlying Vol. Dividend Yield,
Foreign Rate
Interest Rate,
Domestic Rate
Correlation Theta
FEA Delta, Gamma Vega Lambda Rho Eta ðC/ðt
FinCAD Delta, Gamma Vega Rho of foreign rate Rho of domestic rate ðC/ðt/365
Intermark Delta, Gamma Kappa, Vega Rho ðC/ðT /365
Marvin Delta, Gamma Vega Rho ðC/ðt /365
MBRM Delta, Gamma Kappa, Vega Phi Rho ðC/ðt /252
Monis Delta, Gamma Lambda Rho (Div) Rho Epsilon ðC/ðt
Savvysoft Delta, Gamma Vega Foreign Rho Rho corr sens ðC/ðT
Tech Hackers Delta, Gamma Kappa, Vega Phi Rho ðC/ðT
WM Delta, Gamma Kappa Rho_Div Rho, Rho_Int Kappa_corr ðC/ðT

My other seven algorithms produced similar results, which they should, since they are merely different ways to solve the same Black-Scholes-Merton initial value problem. In the limit, as the number of time periods, price intervals and Monte Carlo sample size go to infinity, the algorithms would produce identical results—at least, if the computer made no tiny mistakes such as rounding errors. Of course, in the limit we are all dead.

The different algorithms are not equally fast and accurate, however. The standard Black-Scholes-Merton “analytic” solution is quickest and most accurate. Numerical quadrature can be about as accurate, but tends to be slightly slower. Monte Carlo is slowest and among the least accurate. In a shootout between the pricing algorithms for European calls and puts, the Black-Scholes-Merton algorithm is like a gunfighter who’s downed plenty of coffee, but the Monte Carlo algorithm is like one who hit the whiskey bottle too hard. The two binomial and three finite difference algorithms that I used are ordinarily less accurate than the analytic algorithm and more accurate than Monte Carlo.

Different ways To report outputs

The various ways the packages report and interpret Greeks illustrate the Tower of Babel aspect of the derivatives world. Table 3 lists names for the various sensitivities. The key features of this table are:

  • Nearly all vendors recognize vega as a name for sensitivity to a change in volatility. Three recognize kappa. Monis calls it lambda.
  • Most vendors don’t use lambda at all, but FEA uses it for sensitivity to moves in the foreign rate (dividend yield), and Monis uses it for sensitivity to moves in volatility. So don’t compare the lambdas for FEA and Monis!
  • Tech Hackers and MBRM use phi to indicate the sensitivity to changes in the dividend yield.
  • FEA defines eta, Monis defines epsilon and Savvysoft defines “corr sens” as the sensitivity of premium to correlation.
  • After adjusting for the different definitions for theta, and the fact that FinCad, Monis and Savvysoft report sensitivity to the change in the interest rate or dividend yield, expressed as an APR, the Greeks for the eight packages fell pretty much into line. The results—premium and all Greeks—from my analytic Black-Scholes-Merton algorithm agreed to four decimal places with the results for all vendors. The missing Greeks from some vendors surprised me, however.

The American put

We might anticipate near unanimity in pricing American puts because listed American options and the pricing algorithms have been around for decades. Even after we carefully give each package the inputs it wants and carefully interpret the outputs, however, we find that the pricing for the American put isn’t as unanimous as for the European put. (See Table 4.)

The premium values fell into four groups: FEA and FinCAD; Intermark and MBRM; Marvin; and Monis, Savvysoft and Tech Hackers. This suggests that the eight vendors used essentially four different algorithms for computing premium value. The agreement for the Greeks—after all the usual adjustments for the differences in conventions—was also less than with the European put. In particular, the rho figures are diverse.

In an effort to identify those algorithms, I computed premium and Greeks with eight algorithms. The results appear in Table 5. CRR and JR are the Cox-Ross-Rubinstein and Jarrow-Rudd binomial algorithms. EFD, CN, and IFD are the explicit finite difference, Crank-Nicholson, and implicit finite differences. FEA and FinCAD appear to use Cox-Ross-Rubinstein. Monis, Savvysoft and Tech Hackers appear to use Jarrow-Rudd. Not so obviously, Intermark and MBRM use one variant of CRR, and Marvin uses another.

Table 4 American Put Option Value and Greeks For Eight Commercial Packages
Model Premium Delta   Gamma Kappa Rho_Div Rho_Int Theta BS PDE
FEA 5.7585 -0.4738 0.0280 52.4169 72.3531 -79.9731 -1.1169 -0.0054
FinCAD 5.7585 -0.4523 0.0282 52.4072 80.4539 -71.9449 -1.1248  
Intermark 5.7614 -0.4742 0.0284 52.5461   -80.0527 -1.1227 0.0225
Marvin 5.8717 -0.4995 0.0298 53.6944   -92.7516 -1.1946 0.0000
MBRM 5.5913 -0.4743 0.0284 52.5461 72.3702 -80.0527 -1.1229 -0.0007
Monis 5.7560 -0.4738 0.0283       -1.1059 0.0135
Savvysoft 5.7560 -0.4738 0.0283 52.4800 66.2966 -73.5402 -1.1253 -0.0059
Tech Hackers 5.7560 -0.4738 0.0283 52.4804 69.7095 -77.3243 -1.1261 0.0003
Mean 5.7724 -0.4770 0.0285 52.6530 72.2367 -79.3771 -1.1299  
Std.Dev. 0.0402 0.0091 0.0005 0.4625 5.2255 6.7536 0.0270  

Whether you worry about the discrepancies depends on the use to which you want to put the numbers. Every vendor’s premium and Greek numbers are close enough for government work, which includes value-at-risk and credit risk measures for regulatory purposes. An accountant might not think 1.5 percent of the value of a portfolio was material for financial statements, or even that the overall impact would be that large, after pricing many positions.

A trader might or might not care about model error or algorithmic error that was 10 basis points of notional, depending on his market. Shelly Natenberg, an experienced floor trader, puts the issue into the context of IBM long-term equity anticipation securities (LEAPS): “Because the bid-ask spread in LEAPS can be a point or more wide, whether a market-maker is off by 1/8 or even 1/4 in his quote is probably not significant. What’s at least as important is how he handles the position’s risk once he has it.”

In commenting on currency options, Nassim Taleb, another experienced trader, says that a two-year expiration was “too long,” and that the bid-ask spread for one year might be about “two-tenths of a volatility point—say 16.1 bid, 16.3 offered.” Making all the appropriate changes of variables, I conclude that that amounts to ticking our volatility up and down by just over 7/100 of a percent, which gives us a bid-ask spread in premium of some 7–8 basis points—5.7189 to 5.7931. Compared with this, a model error of 11 or 12 basis points appears to me to be significant.

Call me a worrier, but I believe that the differences are worth exploring, regardless of the market. The spread in premium in Table 5 from the lowest to highest is 0.1125, more than 11 basis points, nearly 1/8 of a percent of notional. If the underlying notional amount is $100 million, the buyer has the trade on his books based on 5.8684, and the seller carries the transaction based on 5.7560, then their net profit and loss is about $112,500 more than it should be.

Practical Suggestions For Buying a Package of Add-ins
I can’t tell you what package to buy, and don’t have the space here to discuss all considerations fully. Here are some key points:
  • Demand that the vendor gives or sells you training—or, at least, sufficient access to a help desk.
  • Speak the language of the package and interpret inputs and outputs properly.
  • Understand your add-in’s algorithm, so you can evaluate its strengths and weaknesses.
  • Make sure the package you buy does what you want and does it right, before you buy it. Merely assuming that it does is not safe, particularly for exotic options.
  • Ask for a demo period of at least two weeks.
  • Make sure you know how to use the package properly, before you start using it in earnest. Most packages include worked examples. Rework them.
  • Check accuracy by comparing results with another package or system, preferably with a user who has more experience than you have.
  • If you can’t reproduce results that you know are correct, and can’t explain the discrepancy—such as different algorithms that differ in a predictable way—don’t buy the product.
—W.M.

What are the correct values for premium and Greeks? The bottom line of Table 5, “Extrapolation,” contains my best estimate.

The European put on a spread

The European put on a spread is a relatively simple and common rainbow option involving more than one risk factor. It has more Greeks than the first two options. It has the usual rho and theta. But with two underlying assets, we have two deltas, kappas and phis, and a sensitivity to the changes in the correlation. The option has three or four gammas, depending on how you look at it.

The dispersion across eight vendors in results for the European put on a spread is greater than it was for the American put—the range is more than 1 percent of underlying notional. This explains part of the large bid-ask spread in the market for exotic options. I attribute this dispersion largely to the fact that the seven vendors attempting to price this product used at least eight different algorithms.

I used three different algorithms to check the premium values and Greeks. The packages produce numbers that are in the ballpark. All the models and algorithms might be correct in the sense that they would approach the correct limiting value if the number of binomial periods or Monte Carlo paths approached infinity, or they might contain some flaws. If a customer wanted my opinion, I would build my own versions of the various algorithms for comparison.

Table 5 American Put Option Value and Greeks For Eight Methods
Model Premium Delta Gamma Kappa Rho_Div Rho_Int Theta
CRR 5.7585 -0.4737 0.0283 52.4162 73.8277 -82.5022 -1.1249
JR 5.7560 -0.4743 0.0283 52.5732 74.6595 -82.5621 -1.1264
EFD (VB) 5.7532 -0.4730 0.0281 52.1109 72.5560 -80.1660 -1.1155
CN(VB) 5.7429 -0.4729 0.0282 52.4014 72.7874 -80.3896 -1.1207
IDF(VB) 5.7330 -0.4728 0.0282 52.6873 73.1506 -80.7632 -1.1258
EFD (C) 5.7532 -0.4734 0.0282 52.4554 73.9629 -82.6173 -1.1223
CN (C) 5.7429 -0.4734 0.0283 52.3505 74.1795 -82.8455 -1.1276
IFD (C) 5.7330 -0.4733 0.0284 52.2537 74.3549 -83.0320 -1.1327
Mean 5.7466 -0.4733 0.0282 52.4061 73.6848 -81.8597 -1.1245
Std. Dev. 0.0101 0.0005 0.0001 0.1785 0.7659 1.1987 0.0051
Extrapolation 5.7474 -0.4733 0.0282 52.3758 72.4851 -80.0658 -1.1208

This glimpse into the world of exotic options tells us that coverage of the Greeks is thinning out. Hardly anybody had cross-gamma, an important measure. Various other Greeks were missing. In addition, everybody has their own way of doing things, and some people have two ways. Consequently, we see larger discrepancies in results.

Summary

Different packages sometimes require their inputs and express their outputs in different forms. Everybody used practically the same model and algorithm to price European puts. Even so, it wasn’t easy to coax the same premium and Greeks out of all the add-ins.

Everybody used a binomial algorithm to price American puts. Six used a variant of the CRR algorithm, and two used the JR algorithm. One add-in had a noticeable bug which the vendor syas is now fixed. Not surprisingly, outputs differed.

Seven out of eight packages could price a European put option on a spread, and three packages had two ways apiece to do it. Five of the packages provided both deltas. Two provided two of three correct gammas and one transposed them. None provided all three gammas. After that, Greeks were sparse.

Learning to use any of these packages properly requires a significant amount of time. I’m not talking about simply loading the package, delivering the inputs, pushing the [F9] key and looking at the results. I’m talking about getting meaningful results that you can use correctly. None of the packages was intuitive for me. Several of them fooled me at first, and one of them took a lot of work to learn to use properly.

The pricing algorithms in these eight packages are like snowflakes. No two are exactly alike, even if they purport to do the same thing—including Intermark and MBRM, which separated only a couple of years ago. This is consistent with my 25 years of experience with derivatives pricing models.

You can’t conclude that one add-in is wrong simply because it produces numbers that don’t agree perfectly with another add-in. They may use different algorithms that agree only in the limit. This problem appears to grow more common with more complex derivatives. I would anticipate a serious problem of this sort with exotic fixed-income derivatives.

Models and Assumptions
The Put Options

I chose European and American puts because they make up a large percentage of open option contracts. I assumed that a system that contains an accurate put option pricing function would contain an accurate pricing function for the corresponding call.

My assumptions about the market conditions of both put options are:

  • The underlying asset is a cash instrument—a share or a foreign currency.
  • The underlying price is 100.
  • Volatility equals 10 percent.
  • Dividend yield (if the underlying is a share) or the foreign interest rate (if the underlying is a currency) equals 5 percent per annum, continuously compounded.
  • The (domestic) market rate of interest is, likewise, 5 percent.
  • The put expires in two years.
  • Its strike is 101.
  • Number of time steps equals 100 for binomial models of American put option premium and Greeks.

The Put On a Spread

I chose the put on the spread of asset two’s price over asset one’s price, struck at 10, because (a) it involves the significant step into a second dimension of risk, and (b) most of the packages claimed to price it. Of course, if the strike were zero one could change “numeraires” and price the option with the Black-Scholes-Merton model.

My assumptions are:

  • The underlying assets are cash instruments—two different shares or currencies.
  • The underlying prices are 101 and 102.
  • The volatilities are 10 percent and 20 percent.
  • The dividend yields are 1 percent and 2 percent.
  • The correlation between changes in logs of the two prices is 0.12.
  • The domestic rate of interest is 5 percent.
  • The European put expires in two years.
  • Its strike is 10.
  • Payoff is Max[0, K -(S2—S1)] = Max [0, K + S1—S2]—a put on the spread of price 2 over price 1.
  • Number of time steps equals 100 for binomial models.
  • Sample size equals 10,000 for Monte Carlo models.
—W.M.

I noticed a tendency for some of these packages to behave like the bird that hatches first from its egg, then tries to destroy its unhatched rivals. The philosophy seems to be “If you capture the Excel menu bar and RAM, then the user’s heart and soul will follow peacefully.” Also, the last one loaded sometimes has an advantage over its predecessors. For example, if two add-ins have the same name, then the last one loaded replaces its predecessor.

Was this information valuable?
Subscribe to Derivatives Strategy by clicking here!

--