Why In-House Systems Fail
Wall Street is littered with poorly designed in-house systems
projects that have been tossed into the digital dust heap.
By Tom Groenfeldt
It's a story repeated over and over again on Wall Street. A derivatives
trading group flush with either ambition or cash decides to junk its outmoded
trading system and install the system of its dreams.
A committee of traders and in-house systems people is convened, an initial
wish list is drawn up and a search of existing systems vendors is assigned.
After a period of research, it's determined that none of the existing vendors
can meet the desk's particular needs. The only reasonable alternative, it
seems, is to develop an in-house system custom-made to meet the company's
A budget is established, people are hired, the clock starts ticking and
money flows freely. A year later, the group reports that several components
are nearing completion and promises that delivery is just a year away. At
least for beta.
Two years later, the system is mired in development hell. Initial designs
did not provide support for later developments and had to be done over.
Traders objected when the look and feel changed from what they had been
using, while risk managers pointed out that the system neglected to provide
them useful information fast enough. Even the graphical user interface has
been changed several times. Meanwhile, several key software engineers have
quit in disgust and gone to vendors or to more technically advanced firms
where they can improve their skills on hot new projects. The team left behind
is decidedly second-tier in both development and project management skills.
To make matters worse, no one on the business side understands software
well enough to evaluate progress, much less direct the program.
Eventually, a consultant is called in to look over the mess. He decides
it's time to junk all or most of the system and adapt an existing system
sold by a vendor. The IT project manager screams about wasting the bank's
huge investment, which he insists is just about to pay off. Afraid of getting
fired, he soon jumps to another firm, and a new cycle begins.
This time, if the firm is lucky and has learned its lesson, it buys a
package to meet 70 percent to 85 percent of its needs, adjusts some internal
processes to accommodate the vendor's design, and then builds the added
functionality it requires in a joint effort with specialists from the vendor
That, anyway, is the picture drawn from interviews with more than two-dozen
vendors and independent consultants, who are called in periodically to clean
up messes like these. The history of Wall Street's in-house derivatives
systems projects, in fact, resembles the repetitive cycles of the movie
At any moment on Wall Street, you can bet that several companies are
wrestling with grossly underestimated time and budget projections, incompetent
systems design, mismanaged resources, unqualified programmers, and constantly
rewritten base code-all jealously managed by competing internal political
groups that are unable to make, or stick with, decisions.
The critical problem is vision. Effective systems development requires
a single vision translated into an architecture. But instead of beginning
with a vision, banks begin with some bullet points. And instead of assigning
responsibility to a benevolent dictator, they settle for a cross between
an American participatory democracy and a Japanese seniority system. To
get anything accomplished requires the skills of Machiavelli.
Most internally developed systems, in fact, start with traders. And that,
say consultants, is where the problems begin. "Almost every in-house
system originates with a bunch of traders getting together and writing a
quickie," says Jos Stoop, president of Intuitive Products International
Corp. "If you build that out, you will run into something that needs
replacement. Typically, the heart of the system is seen as the stuff the
traders and analysts come up with. Unless you pull that apart completely
and reassemble it in a modular fashion, it is difficult to get away from
In most cases, traders want systems that meet their immediate needs,
and they want delivery yesterday, at the latest. Their self-confidence is
rarely tempered by any trace of patience. When IT doesn't deliver, traders
are apt to spend a weekend putting together a simple program themselves
and loading it Monday, without telling the wet blankets from IT. "In
every bank," recalls one consultant, "traders take pride in what
they have accomplished. The systems people say it is awful and create their
own. Then traders say they can't work with what systems people built."
"It's not that systems people simply don't get along with traders
and the business side, it's that they hate each other," explains another
former trader who now works for a derivatives systems vendor. "They
don't talk, they don't cooperate, they aren't even organized along similar
One thing traders don't understand is systems architecture-the idea that
front, middle and back offices have different functions and different requirements.
A trader-developed trading system, for example, might have good front-office
analytics running in real time but may not be able to perform credit risk
management because the middle and back offices use batch processing. Risk
management, after all, is nothing without linking current trade inputs with
data from legacy back-office systems to provide up-to-date exposure information.
Traders are notorious for losing interest in a trade after its input, but
if the information can't flow smoothly from front office to back, risk management
Fixing these shortcomings inevitably leads to patches, add-ons and workarounds,
but never to elegant design. Although banks boast about their global trading
books and comprehensive risk management, many of them still transfer their
order information by e-mail. "Financial institutions developed their
systems in step with their activities in the market," says Simon Moss,
head of the Americas trading and risk management practice at IBM. "Many
of the solutions were robustly built but were often designed for specific
functional coverage. They rarely took into consideration demands for such
things as integrated, consistent firm-wide risk management or horizontal
client-rather than product-based structures."
Consultants say that one Swiss bank spent $3 million to $4 million trying
to build an in-house swaps system running on two IBM 3090 mainframes before
it finally relented and bought SunGard's Devon. "The problem arose
from starting to build a piece of the system without designing an architecture,"
says one consultant. "When banks decide to build a system in-house,
they start talking to users, but the system is always driven by the needs
coming from the front office, which worries about pricing. Only later do
designers begin to think about risk management and credit management."
Equally disastrous is asking everyone what they want from a system without
having a method to cut the project down to the essentials. "Firms fall
into the toy store syndrome," explains a consultant. "Everyone
wants something, so they decide to write specs." One Swiss bank did
just that, and $35 million later, developed the specs for a great system
but nothing else. Toronto Dominion, for example, also reportedly spent $4
million to $5 million by first asking everyone for a wish list. It quickly
had a massive wish list, and three years later there was still no software
to show for its ambitious plans.
Although traders get their fair share of blame, the blame falls equally
on in-house systems managers, who often push big projects because they provide
them the opportunity to build up their fiefdoms. "These projects take
on a life of their own, and banks throw good money after bad," said
one consultant who used to work in a bank. "Software engineers will
build a word processor from scratch, just to learn how to do it, rather
than just going over to Comp USA and buying one for $99," said another.
This do-it-yourself-at-all-costs mentality is exacerbated by basic inexperience
with the way systems components must work together in a larger architecture-a
lesson that vendors often learn by painful experience. One Japanese firm,
for example, recently completed a high-tech global risk management system
whose performance is bogged down by its graphical user interface. "It's
very difficult to change anything on the screen," said a consultant.
The GUI, chosen because it is multiplatform, turned out to offer more flexibility
than the bank needs, since it is running just one type of workstation.
Another common failing is a basic failure to understand the limitations
of their own designs. The New York office of a French bank has a derivatives
system that was perfectly suited for a boutique business. But now that volumes
have grown, a database that crashes somewhere beyond 5,000 deals is proving
to be a disaster. When it takes hours at the end of the day to do revaluation,
the system becomes completely useless for intraday risk management.
These design mistakes are often compounded by poor project management
decisions. The very top firms, such as Citibank and JP Morgan, have IT staff
capable of undertaking complex development programs, says one London consultant.
"But the next tier down, that's not necessarily the case. It is getting
to be much more complex than many banks have the resources for. In many
cases, the staff at the bank has no project management experience or no
experience with the technologies involved."
And when banks do manage to hire a competent systems team, they typically
underestimate the complexity of a large development project. "They
think that throwing 100 people at a problem should take care of it,"
says a software expert at a small shop, "but running a large software
project requires extraordinary skills that only a few companies such as
IBM, Lucent and Martin-Marietta have really mastered."
IT departments taking on development projects to keep abreast of new
technology often do it backwards. They start writing data classes and lower-level
parts of the system while they are learning the technology. By the time
they move to higher-level development, their early building blocks are outdated,
so they have to go back and modify them. The result is kludge piled upon
kludge. "This type of project doesn't lend itself to teams doing it
the first time around," noted a consultant. "It would be more
useful to buy something and work from there, but that is not in the IT interest."
The project management problems are compounded by an ugly truth: most
top banks can't attract enough top programmers. One highly skilled programmer
is more productive than several moderately skilled ones, and there just
aren't enough great programmers to meet the demands within the financial
industry. "It's hard to find C++ programmers on Wall Street, and even
harder to find quants," says a vendor who has a few of both on staff.
Software experts who understand the business challenges of derivatives are
extremely rare. Perhaps it shouldn't be surprising that once a system is
built, the owners often aren't sure what they have.
When they do get good talent, it's hard to keep them focused on long-term
development projects. Software engineers at banks don't want to spend years
on a single project. It gets boring, they risk abrupt termination if the
project isn't accepted, and their marketability decreases as they limit
their work to the narrow scope of a single derivatives project. The best
engineers want to work on advanced projects with other experts. Consultants
say one Canadian bank saw its derivatives development grind to a halt as
its IT staff drifted off to other jobs. Threats can come from unexpected
directions. At another Canadian bank, a software team was working long hours
on development. During those long hours, two key team members became romantically
involved and then left to get married. The project died.
Another common project management failing is mismanaging the work assigned
to different teams of outside vendors. One bank, for example, decided to
bring in two vendors so it could develop both a middle office and a back
office simultaneously. Not surprisingly, the solution was very slow. Equally
predictably, each vendor blamed the other.
One outside consultant found the bank couldn't even cooperate at the
database level but was using intermediary views. "The bank had no control
over what these people were doing," explains Intuitive's Jos Stoop.
"After more than two years of work, no workable solution had been produced,
so they called us in." Stoop found the bank was blaming the wrong vendor
and suggested improvements that made an unusable result into a crippled,
but working, system. "A lot of it was forcing the vendors to work together
because they had both been hiding by blaming each other. You cannot leave
it up to vendors to do the right thing for you."
Fear of commitment
Dealers shouldn't be surprised if their systems groups underperform or
if morale is low. Many banks don't have the patience to invest in long development
programs with no visible returns. In many cases, the higher-ups call out
for human sacrifices, although programmers will often suffice. Basically,
says the vice president of a large financial technology firm, banks have
begun to realize that software development is not their core business. "At
some point, even big banks like JP Morgan will realize that they have staffs
costing millions to support proprietary software. They have to maintain
it and staff a help desk, and that isn't their business." Eventually,
even banks with tens of millions invested in a functional home-built derivatives
system will find the courage to shut it down.
When a bank decides to cut its development budget and has to decide between
a small project producing usable programs and a three-year development project
that won't produce a useful program for another 12 months, there's no question
of what is going to get axed first. All this is unlikely to inspire great
loyalty among software staff who know their careers depend on their ability
to remain current with the latest programs and tools.
Among consultants and vendors, the most talked-about recent development
failure is at a large British bank. Details vary widely from one account
to another, but depending on whose tale you believe, the bank spent somewhere
between $10 million and $80 million on a system that "ultimately doesn't
do anything," according to one former bank employee. For several years,
a British executive in the New York office of this bank drove development,
but years of development weren't enough to overcome an extremely convoluted
design. "Everything got broken down into metaphysical components,"
explains the former employee. "They abstracted too far from the nuts
and bolts of what they were doing."
In typical bank fashion, the development continued with little or no
accountability as long as the program director remained in political favor.
Eventually, however, senior management changed its opinion of him. He left,
and the development project has died, leaving the bank with a ticket-writing
system and some word processing capability in return for its multimillion
dollar (or pound) investment. Some consultants say the bank is about to
buy Summit Systems. More recent accounts say the bank plans to try again
with in-house development. The bank, several consultants agree, has gone
through a series of management changes, each of which has changed its development
At the end of many exhaustive development projects, it's not even possible
to determine whether the project was a success or a flop. Citibank, which
spent upward of $30 million on its Global Derivatives System, had to hire
outside consultants to tell the bank whether it was a success of failure.
The project was over budget and past deadline-not an unusual result in a
complicated software scheme-so critics were calling it a disaster. The consultants'
verdict? The project was officially designated a success. Give that team
a bonus-if they haven't all been fired.
Other times, even when a system works, it is at least partially a failure,
says one consultant who worked on a Swiss bank's project. Originally planned
for 1218 months, the system took three-and-a-half years and was multimillions
of dollars over budget. "Size alone can be a big problem in a huge
system. Time eats up enthusiasm, [and] it's difficult to hold onto the professionals
and to maintain the budget."
The ultimate problem, say consultants, is that banks simply try to do
more than they can deliver. Banks commonly bite off more than they can chew,
says Josie Palazzolo at Sailfish, a risk management software developer.
"They are looking for a big bang approach, and in risk management you
have to do it in small stages. There is often no right or wrong answer-there
is a best answer for the amount you are willing to spend in the time allotted
for the problem."
Why Vendors Have An Edge
By Tom Groenfeldt
Vendors will eagerly tell you they can provide better software than banks
can develop for themselves in-house. They also claim they can deliver it
faster, and often even provide documentation-something often overlooked
during in-house system development. What vendors are less likely to tell
you is that they are more likely to do this because they have learned from
their own repeated mistakes and failures. Their lead over mere banks is
that, having been through the error cycle several times, they have already
learned what not to do.
When banks try to build software, they are systematic and rigid, says
Gabriel Bousbib, vice president of the risk management unit at Reuters.
They start with a clean slate, plan to build the best system money can buy,
hire lots of people and try to meet all the desires presented by all business
units. What you need, he explains, are just a few key objectives, and a
small team of smart people commanded by a dictator. Then lock them in a
room and let them work.
One key thing vendors have learned is the importance of getting the architecture
done first. Again, that's experience speaking-most software shops were founded
by developers who left banks in frustration and set out on their own with
strong beliefs about how a system should be built and little appetite for
taking orders from someone who doesn't understand technology but has greater
rank in the hierarchy.
Vendors manage to avoid long drawn-out projects is pure self-interest;
they'd go broke. Vendors that manage to survive do so by developing an architecture
that usually reflects the vision of just one or two people-the firm's founders.
Then the company tries to build modular and reusable pieces it can sell
"Design everything and implement a little bit," advises John
Fry at Principia Partners LLC, a risk management software house in Jersey
City, N.J. "Banks are not in the business of writing software,"
says a consultant who gets called in when banks or software houses get bogged
down. "They should let vendors develop the basics-GUIs, replication
and data integrity. Where they get the most bang for the buck is adding
an API or using a toolkit approach."
The movement toward open systems strengthens the case to buy rather than
build. Individual software companies can develop specific products in-depth
on a standard platform. Then the products can be linked by systems integrators
or on Windows NT by Microsoft's Object Linking and Embedding (OLE) technology.
"Having a common infrastructure will change the dynamics of how banks
mix software," says Reuters' Bousbib. "They will buy components
and plug them into a common infrastructure." Then bank IT staffs can
add proprietary models to achieve customized systems.
"Banks want a system that can handle the entire universe of products
or provide a handoff to a specialized system like Sailfish or Algorithmics,"
says another vendor. Banks shouldn't expect to develop a system for less
than vendors spend on their development programs, said one vendor. In fact,
since they are starting fresh, it may cost them more. Infinity, which went
public last year, calculated its research and development costs at $6.1
million in 1995 and $4.1 million for just the first six months of 1996.
When the margins earned by derivatives groups were several hundred points
higher than they are today, it was relatively easy for a bank IT department
to make a case for building an expensive system. Now, however, derivatives
often generate profits comparable to fixed-income desks-but the supporting
systems can cost 10 or 20 times as much. In this new environment, the business
case for building expensive in-house derivatives systems becomes difficult
Despite all the advantages third-party vendors possess, systems consultants
caution that third parties can rarely provide a completely successful front-to-back-office
solution. "The success of any project involves synthesizing the internal
and external components," says IBM's Moss. "If you run everything
in-house, the project could become unfocused as internal pressures affect
the project. It could also become undisciplined as people ignore the project's
original methodology. But if you give too much to the vendor, the client's
original requirements may get lost or misinterpreted. The ideal balance
involves close control and scrutiny of the project's direction by both users
and the IT group."