Deciphering Risk Reports
What’s the good of a fancy risk management system if nobody understands the results?
By Tatiana Helenius and Tom Groenfeldt
All the talk and money devoted to the development of high-end risk management systems misses an important point: Most value-at-risk and other risk management reports are so clumsy, awkward and difficult to decipher that few of their intended recipients fully understand them.
In many cases, senior managers receive their reports, develop a glazed expression and then call their risk management staff. “Unfortunately,” says Christopher Hamilton, a partner in KMPG’s Financial Services Practice, “many risk reports are still a bundle of pages.”
Because these risk management reports are so impenetrable, they are often ignored when it comes time to make critical decisions. In many cases, the problem is that senior managers do not know what risk reports are trying to convey. Although they may be able to identify something as a VAR report, they may not know what VAR means, and how to use this information within a strategic context.
To make matters worse, risk reports—even those developed by major systems vendors—are not particularly interactive. Even if a risk manager was able to identify the problem, he or she may not be able to trace an anomaly back to its source easily and accurately. “What’s been missing has been any interactive use of risk management systems by anybody outside of the risk management operations group, including senior management,” says Peter Davies, President of New York-based Askari.
For firms that have spent millions of dollars on enterprise risk management system implementations, this inability to extract reports in some usable form is a major obstacle to realizing any return on their investment. “Vendors have had the luxury of pretty well ignoring the reporting function,” says Debbie Williams, a principal at Meridian Research, who estimates that reporting constitutes less than 2 percent of the total amount spent in enterprise risk projects.
Recently, however, there has been a movement afoot, both within financial institutions and at software vendors, to supply risk management reports that users can actually read at a glance, interact with and use to glean some useful information.
Currently, more comprehensible risk reports can be found in a number of places. First, the large risk management systems vendors are working hard to make their often incomprehensible reports easier to understand. Second, other vendors are offering do-it-yourself analytic reporting tools with better-than-usual graphical capabilities. And finally, many firms are choosing to distribute risk management information in lowest-common-denominator form, downloading precalculated risk figures into Excel and PowerPoint templates.
In the end, however, selecting the most effective presentation vehicle for risk management reporting is a subjective decision. Beauty—and clarity—is in the eye of the beholder.
It seems self evident, but before you can create a totally comprehensible risk management report, it is necessary to identify what, in fact, constitutes a “good” risk management report. Here are some criteria that provide a reasonable starting point:
The right report for the right person. There are many different constituencies within financial institutions who need to see risk numbers, but the level of detail and explanation varies greatly depending upon who, exactly, will be looking at the report. KPMG’s Hamilton describes four tiers of risk report consumers. These include “front line” traders, who require real-time information and a great deal of detail; risk managers, who require summary information on specific trading books and so on, but also require the ability to drill down; senior executives, who need firm-wide strategic analyses and value-at-risk reporting; and regulators, who require reports according to their own specific formats.
Because each of these different constituencies has different reporting needs, there is no such as thing as “one size fits all” when it comes to risk management reporting. Different strategies are required to create reports that speak to each group.
According to Teresa Adams, a partner at Principia, developing appropriate reports for risk managers in particular can be difficult because, in many cases, risk management reports are drawn directly from trading systems. “You have to ask who is using the trading system, who’s designed the trading system and whose needs that trading system addresses. If risk managers are supposed to supply independent verification, can they do that if they are using the same system that the trader is using? Do they have a means for independently assessing and managing risk? That’s something that’s often missed.”
Visual benefits. The second criterion for risk management reports is that they should be understandable “at a glance.” Most often, this means finding some way to visualize complex numbers in the form of a graph or other sort of chart. “Creating visual representations of complex risk management numbers is key,” says Andrew Bruce, a senior research scientist at MathSoft. “You can say a lot more with color and shape than with pure numbers.”
|“Vendors have had the luxury of pretty well ignoring the reporting function.”
And, of course, visual displays must be effective. This often means creating a consistent format, using colors to indicate change and different magnitudes of change. “Traffic lights use green, yellow and red to convey specific meaning,” says David Gilbert, president of C-ATS Software. “If there is a policy violation, or a risk number requiring immediate action, we need to use a color that is extremely attention-getting, that users associate with ‘Warning, Danger Ahead!’”
Multiple dimensions. An-other aspect of risk reporting is the ability to convey the multiple dimensions of risk in a clear, straightforward manner. “It is essential to provide multiple views of risk without resorting to numerous, mind-numbing tables,” says Chuck Jones, senior product manager for BARRA. “One way is to present key data graphically across multiple axes. The graphical approach is useful, because it draws the eye to major exposures.” Another approach, explains Jones, is to custom-tailor risk reports so that each decision-maker receives information that is more relevant to the specific dimensions of risk that he or she cares about.
Staying in context. Risk management reports must also include some sort of information to place graphs or tables in a meaningful context. This context might include similar data from the recent past, such as a graph plotting today’s VAR vs. VAR over the past month—and a brief explanation of the assumptions underlying the calculations at hand. Context also includes clearly explaining what, exactly, a report is trying to convey. If the report is a scenario analysis, it should explain the scenario; if it is a sensitivity report, it should explain which variables are being tested, and specify the time period.
Systems that are specifically designed for risk management usually avoid the compliance pitfalls of culling risk information directly from trading systems. Many of the largest risk management vendors have been criticized for their impenetrable reports, however. This is only logical—most of these firms were founded by quants, who started with the mathematics of risk management.
“Risk management systems, unlike trading systems, have been chosen for their ability to do two things: aggregate data and provide a lot of computation on that data,” notes Askari’s Davies. “The primary selection criteria have been the ability to interface with other systems, warehouse data, handle volume, provide sophisticated models and so on. The reporting requirement has not been emphasized because it’s been secondary to the mechanical exercise.”
But problems begin to surface when end-users realize that their costly risk management systems were being used only by a handful of quants. While the analyses these systems can provide are complex, many of the presentation issues these companies are now addressing are long overdue.
Perhaps the most prevalent feature that vendors are adding to their risk management systems to make the output more accessible to a wide variety of users is flexible report design. “One of the best things you can do as a software supplier is provide easy-to-use report design tools, in addition to the usual canned reports,” says Raj Patel, a vice president at Integral Systems. “This lets users create reports that deliver exactly the information they need.” A user-configurable report writer, then, is an important way of ensuring that every user has the reports that are right for him or her.
Flexibility also entails the ability to access the most current data at the most convenient time, says Alex Tsigutkin, president of Axiom Software Laboratories—particularly when a risk manager is accessing information while on the road, or happens to spot a potential problem that needs more investigation. “Ad-hoc functionality is important in risk management reporting. There are, of course, standard reports that are available daily, on an intraday basis or at some other level of frequency. But the ad-hoc function is also important for when risk managers need to research quickly what is taking place, or need to be able to construct search criteria for the reports themselves.”
In many cases, however, end-users are not even getting the maximum benefit from their vendors’ existing risk-reporting features, some of which allow a great deal of leeway for flexible report design and graphic development. The reason: a communication gap between those who implement the risk management system and those who actually use it.
According to C-ATS’s Gilbert, the first step to developing coherent presentation is for vendors and the consultants or IT staff who are handling a risk management systems implementation to ask their clients what information they’d like to see, and how they’d like to see it. “Developing good presentation is an iterative process that should involve the end-users of risk management reports,” he says. Often, this process may involve creating sample reports and display formatting, which users can critique.
Another issue risk management vendors are tackling is displaying extremely complex analyses, such as Monte Carlo-based VAR, multidimensional scenario analyses and so on, which may involve literally hundreds or thousands of pieces of information. According to Hugh Richards, a product manager at Infinity, “Graphs, charts and other visualization techniques are an intuitive way of providing a rapid understanding of complex risk management analyses. Simply put, if your graph is smooth, it indicates you’re doing your job; if it’s spiky, you’ve got problems.” Adds Patel, “The more graphics you can use, the more quickly users are going to understand what is happening.”
More frequently, risk management systems are offering heat maps as a way to visualize complex, three-dimensional reports in an easy-to-understand, visual way. Heat maps generically refer to any graph hat uses a warm color in greater and lesser intensities to indicate where the hot spots are.
That kind of visual summarization makes it easier to eyeball reports and zero in on major changes in minutes. Most risk managers, says Dan Rissin, one of the principals at TruRisk, don’t want to spend the bulk of their time wading through information that is basically static. “In general,” he says, “yes, you want to know what your risk is. But more specifically, you want to know what has gone up since yesterday, what has gone down, what is historically high, what is trending up or trending down. Any capable manager should be able to spot those changes right away. They need to be able simply to look at the report and say, ‘Ok, we’re only 70 percent of our limit on this thing, not too bad.’ Or, ‘We’ve gone 10 percent above our limit. Let’s find out what the hell’s going on.’”
The key is using configurable graphics, and honing in on active elements in the report. Axiom’s Tsigutkin illustrates one example drawn from energy risk reporting. “In an energy business there are different dimensions for each kind of risk factor. You have different locations, commodities, time zones, “seasonalities.” You might have to span correlation metrics of 5,000 factors by 5,000 factors in a single report. It’s a challenge. But if you want to find out why the risks have moved between yesterday and today, you can just print those elements of your correlations, metrics that have moved since yesterday. In that case, you’ll be able cover all the elements much more neatly.”
|“You have to ask who is using the trading system, who’s designed the trading system and whose needs that trading system addresses.”
Not everyone, however, agrees that graphics are an important aid to understanding risk management figures, says David Penney, CTO of Algorithmics. “The graphic content isn’t really that important. It is really much more important to define what numbers you are producing and not giving too many numbers. If you need so many numbers that you need some kind of a visualization to display it, chances are that you are doing the wrong job.”
Kelsey Biggers, principal at Micro Modeling Associates, says he’s found some of the visualizations of risk completely confusing. “One report had so many columns sticking up out of it, it looked like the Manhattan skyline,” he says. “I couldn’t discern what it was trying to tell me. It might work for the guy on the desk who will look through several hundred categories, but senior management might want to see five categories at most.”
While aggregate reports are important, particularly to high-level managers, it is equally important that users be able to “drill down” in order to investigate further how aggregate numbers have been produced. This can be accomplished in a number of ways. Aggregates can, for example, be displayed at the top of the table, which then lists the various components of the summary number. This approach may be effective for position reporting on a small trading book, or for high-level roll-ups, such as total, global foreign exchange positions, which are then broken down into, say, their regional components.
|WHO SHOULD GET WHAT REPORT?
|How much/how quickly
||•Book-level position reports
•Book-level P&L reports
•Single deal analysis
•Market factor analysis
•RAROC, risk-based limits by individual
|•Trading systems, including separate front-office modules
•Integrated front and middle offices and spreadsheets
•Want as much information as possible
|•Keep it simple
•Let traders use desktop displays to view info à la carte
|•Avoid small fonts
•Avoid unnecessary graphic content, which slows down the system
|Quants/new product development
•Market factor analysis
•Some risk management packages
•Custom analytic systems/develop-ment tools
•Want information pertaining to specific questions/ problems
|•Use multi-dimensional/“3D” displays to capture complexity
•Build in flexibility to design individualized reports
|•Avoid canned reports or analyses
||•Firm-wide, department-wide position, P&L reports
risk-based trading limits
|•Risk management systems
•Reports from trading systems
|•Daily, intraday information
•Aggregated summaries that allow user to drill down to anomalies
|•Minimize numbers per page
•Communicate problem areas (spikes) immediately
•Allow risk users to drill down to various tiers of detail
|•Redesign trading reports to support the risk management function
|•Risk management systems
•Decision support systems
|•Daily or weekly information to be used for enterprise-level decisions
||•Explain clearly what the report describes
•Keep everything as simple as possible
|•Avoid reams of numbers, complete with equations for reference
Another way that vendors are trying to enhance their reporting capabilities is by providing interactive drill-down capabilities, in which users can click on a specific point on their report, which then automatically provides the next level of detail, explaining how that number was arrived at. “We provide software tools that navigate deeper and deeper levels of detail,” says Chuck Jones, a senior product manager at BARRA. “Nowadays, that’s pretty standard—users want to know what their reports mean, and exactly how high-level information is generated.”
Getting out of the application
Another option for creating clearer and more meaningful risk management reports is to push more information out of risk management data warehouses and into other programs designed both to analyze these data and display them in a visually compelling format. Analytic packages, such as MathSoft’s S-PLUS and the MathWorks’ MatLab products, are development toolkits that include some state-of-the-art data-display capabilities along with analytic tools. While nobody in their right mind would build an entire enterprise-wide risk management system with either product (too slow!), some firms have taken to downloading specific information into S-PLUS and MatLab applications to run statistical models that their risk systems cannot accommodate and to run attractive reports. Indeed, many users simply go straight to the desktop, downloading risk management reports—or even re-inputting them—into PowerPoint and Excel spreadsheets to make them more accessible to users.
|“If there is a policy violation, or a risk number requiring immediate attention, we need to use a color that users associate with ‘Warning, Danger Ahead!’”
Some system developers are convinced that the trick to the best kind of risk reporting will involve taking advantage of the Internet by boosting interactivity and communication from the trading level through to the upper echelons. Richard Walker, director of product management at Infinity, suggests “wrapping up risk engines using Java, then making the output of those reports accessible from a web browser. Allow people to customize exactly how that report would look in the browser, in terms of what information will be presented, how it will be presented, whether it’s tabular, a three-dimensional, a two-dimensional chart, what colors are going to be used in terms of warnings, what labels, what descriptions, how these things are to be laid out on the page.” Once a report is created, people should be able to use a standard off-the-shelf browser, point it to the appropriate page and get the information they require. Risk managers can then interact on-line with the report in order to make comments and requests, or to flag items for attention.
The goal of using browser technology to give everybody the risk information they need looks like the most reasonable way to make the risk management process truly relevant to the institutions it is trying to serve.
|Keep It Simple, Stupid
|One more risk management report hits your desk and the world goes black. It might be all those numbers. Like a vast chocolate box— tons of selection, with not an idea where to find what you are looking for. Do you know what most of these figures mean to you? Probably not—unless risk management system developers quash the quant-speak and translate the information into the language of business.
The problem with many risk management reports, says Benny Cheung, a vice president at JP Morgan, is that “you have product specialists reporting the risk position of their speciality.” In other words, your option specialists report their option risks in terms of the greeks—the deltas, gammas and thetas. Then your fixed-income specialists categorize their risks in terms of durations and convexities. And people in the equities field may prefer to look at their risk in terms of how closely it correlates with the underlying stock market indices. “Each method has its own merit,” says Chung. “But imagine you are a manager with jurisdiction over all three products. You would have to learn three languages.”
Taken together, the information might be useless for making informed decisions if you’re forced to compare apples with oranges. The best solution may involve selecting one risk measure capable of capturing risk for a wide variety of instruments. Looking at risk from the management level forces you to aggregate different risk measures. That’s why a lot of people prefer something simple, like VAR, for that purpose.
Cheung recommends summarizing the important information as much as possible.
“Your risk report summary should be short and concise, and should fit in a short paragraph,” he suggests. “Then people with tight time schedules, especially senior management, can spend 10 seconds and judge ‘Should I look at this right now or should I look at it in a few hours time—or even tomorrow?’ Summary is essential. The reader should not be inundated with details. The information presented initially should be just sufficient enough to allow people to ask the next question.”
In this context, visual detail is not just cosmetic—it’s one of the best ways to spot problems right on the surface. Heat maps get to the point particularly quickly. To plot your credit exposures enterprise-wide, you might have a world map graphic with high-risk areas in red, and gradations of color representing decreasing risk, down to, say, blue, on the opposite spectrum. “Southeast Asia, including Japan, would be on the red side of the spectrum right now,” notes an analyst. “This gives you an immediate impression of where problems might arise. The merit of this kind of visual reporting is that it is easy to remember—once you’ve seen it, it sticks in your mind; you don’t need to be a rocket scientist to understand it at once.”
|Was this information valuable?|
Subscribe to Derivatives Strategy by clicking here!