Birth of the S&P Managed Futures Index

In recent years, managed futures have been one of the more interesting "niche asset classes" for diversifying traditional stock/bond portfolios. Managed futures encompass a category of hedge funds in which Commodities Trading Advisors (CTAs) develop trading strategies using futures contracts. For several decades, managed futures benchmarks have demonstrated CTAs' ability to produce attractive returns with fairly low correlation to U.S. stocks and bonds.

In January of 2002, S&P launched its Managed Futures Index (MFI) as the first "build-out" of components included in the larger S&P Hedge Fund Index (HFI). You can read my column about the background of the HFI here:

• Indexed Alternative Investments Are Here?And Getting Better

With the launch of the MFI, S&P published a methodology white paper explaining the index's construction and five-year pro forma (back-tested) performance statistics. You can read the white paper at:

• www.standardandpoors.com

Click on "Indices," then "S&P Hedge Fund Indices," then "S&P Managed Futures Index" and then "Methodology."

The pro forma MFI performance record spanned 60 months from 1998 through 2002, and the actual performance of the MFI has produced a 25-month track record since. Is this performance record one you should share with your clients who are interested in managed futures? Let's look deeper, because doing so sheds light on how track records are being created and why standards have become essential.

First, recognize that despite a stellar reputation for objective analysis, S&P is not exactly a neutral observer of hedge funds or managed futures. S&P is a profit-making division of a pubic company, McGraw-Hill, and it has created these indexes to earn licensing revenue. In my previous column on the HFI, you can learn about the "exclusive perpetual license" that S&P has granted to PlusFunds Group.

S&P has been a profit-making entity for a long time, and yet it has always managed to maintain objectivity and credibility with investors. But in designing the MFI and creating its pro forma performance record, S&P may have crossed an invisible and murky line that separates objective analysis from caveat emptor. I'll tell you the facts, you can check them yourself, and then decide. Remember: The real point isn't S&P's ethics because they are universally considered to be beyond reproach. The issue is: What will you personally accept as a performance record to show clients? If you can't accept the data S&P has published, for reasons I'll suggest, it should make you wonder about performance data in general.

The Pro Forma Period (1998-2002)

In its MFI methodology white paper, published in March of 2003, S&P documented a sterling back-tested pro forma performance record for this new index, one that would make almost any investor take notice. It is summarized below.

The white paper was not shy about summarizing these results. It said: "With an annualized return of 13.7% for the five-year period, the S&P MFI considerably outperformed the managed futures and other asset class benchmarks."

In further statistical analysis, S&P highlighted two other benefits that were important to investors in the dark stock market days of early 2003, when the MFI was launched.

Pro forma results showed that the MFI had a strong negative correlation of -0.45 with the S&P 500 Index. No other asset class except gold had ever demonstrated such ability to offset the volatility of U.S. stocks.

• Through the worst "down months" in the U.S. stock market, the MFI had demonstrated solid performance. During ten months when the S&P 500 Index lost more than 5% from 1998 through 2002, the MFI was positive in all but one. In six of those 10 months, the MFI was up more than 5%.
• If you had shared this pro forma performance record with your clients, surely they would have been convinced that the MFI had shown strong ability to perform well while offsetting downside losses in U.S. stocks.

MFI Real Performance (2003-2005)

The MFI has now had 25 months to show that the pro forma claims are real, and thus far it has not measured up. During a period when the S&P 500 Index has increased by an annualized 16.4%, the real MFI has returned just 4.4%. The correlation of the MFI with the S&P 500 also has changed substantially in real results (+0.26) compared to pro forma (-0.45). Since March of 2003, there have been five months in which the S&P 500 Index return was negative. In all five, the MFI was negative, too, and the MFI's cumulative loss in those five months was more than twice as great as the S&P 500′s.

So, what changed? Why is the gap between pro forma and real so great? You have to look closer to see some reasons. When you do, you will understand why it is critical to set personal standards for acceptable performance records. You also will see what those standards should be.

Looking Closer at "Performance"

The MFI was not the first index of managed futures performance, but it was possibly the first such index specifically designed to attract large numbers of investors to this relatively unknown asset class. The tip-off was in the variance between the MFI pro forma results and more traditional benchmarks. For example, the Barclay CTA Index, which includes more than 300 CTAs, produced only about half the MFI's return and volatility during the five-year back-test period. Over longer periods, the Barclay index had demonstrated correlation with the S&P 500 that was near zero (neutral), not negative.

• http://www.barclaygrp.com/indices/cta/sub/cta.html

Another broadly-based index of CTA performance is maintained by Managed Account Reports (MAR) and called the CISDM Trading Advisors: Qualified Universe Index. Monthly performance for this index dating back to 1980 can be accessed here:

• http://cisdm.som.umass.edu/indices/trading/ctaqualifieduniverse.asp

Analysis of this extensive data base clearly shows that during the 1990s managed futures became a larger, more competitive, and more diversified industry in which aggregate returns showed more stability and less volatility. The same trend was documented by the Barclay index, as shown in the table below.

The MFI performed better than the established benchmarks on paper because it was a different kind of index-narrowly based and non-diversified among CTA styles. Among hundreds of qualified CTAs that could have been included in the MFI, S&P selected just 14 index constituents, all of which follow systematic technically-driven quantitative methods. Many other more intuitive or "discretionary" managers included in the Barclay and CISDM benchmarks were excluded from the MFI. This increased the MFI's pro forma return (above the industry's broadly-based benchmarks) while also creating an index more volatile than the managed futures industry as a whole.

S&P said that a narrow group of index constituents was necessary to create an "investable" index. But is it a coincidence that this particular group of 14 CTAs had outstanding performance and a strong negative correlation with the U.S. stock market, during a pro forma period that included a deep bear market? Surely it is not a coincidence, as S&P acknowledges in this telling comment from the MFI white paper:

"Past performance of the pro forma index is based on back-tested results that do not represent the results of actual trading but are achieved instead through retroactive application of an index strategy designed with the benefit of hindsight."

Here's one way to translate that: Anybody can design a high-performance track record on a back-test basis by cherry-picking select funds, managers or securities. The more narrowly focused a strategy is, the easier it is to cherry-pick.

The downside to cherry-picking pro forma performance is the ensuing real results, which can't select top performers with hindsight and usually falls short of pro forma. This apparently has been the case with the MFI. The high-performance, high-volatility, non-diversified focus that produced the MFI has, in the real world and a changed market environment, become its nemesis.

Other Pro Forma Problems

Cherry-picking isn't the only problem demonstrated by the MFI. Here are a few others:

• Pro forma performance data was supplied by the constituent CTAs themselves, not independently verified by S&P. It is impossible to assess accuracy. (The real MFI results are independently verified by a third-party administrator, Derivatives Portfolio Management.)
• The pro forma data is not comparable to "real world" data because it doesn't include loads, brokerage fees or commissions. As you can see in a tiny white paper footnote: "If [these costs] had been factored in, the pro forma index returns would have been lower."
• The same footnote discloses that some CTA constituents included in the index did not provide returns for the five-year full pro forma period. However, S&P does not disclose which CTAs did or did not provide five-year records. Obviously, not including comparable back-test periods for all constituents (and not disclosing details) increases the potential for cherry-picking.

When you look at the MFI's pro forma performance record in light of its actual performance, and also the longer track record of more established managed futures benchmarks (e.g., Barclay's and CISDM), you may be saying: "Something is wrong with this picture. Where's the accountability?"

Yet, here's the most amazing point of all: S&P publishes an MFI fact sheet updated monthly, which may be downloaded from its Web site. To this day, the fact sheet publishes statistics that combine pro forma and real data into one performance record stretching from 1998 to the present. For example, the most recent fact sheet tells you that the MFI has correlated at -0.34 with the S&P 500 over this six-year-plus period.

It's not enough that many investors believed the MFI pro forma story in early 2003 and paid a price for doing so. Without a compelling real track record to stand on, S&P is encouraging today's investors to keep believing the same story by mingling fiction with reality. You have to wonder: When will S&P let the MFI's real track record stand alone and be judged on its own merits?

Five Standards for Acceptable Performance Standards

You can't rely on anyone else to define an acceptable performance record to show your clients. In today's industry, you must set the bar yourself. Much of what we have learned from the MFI can also be applied to performance records of mutual funds, separate accounts and hedge funds. The standards that will protect you and your clients are as follows:

1. Don't show pro forma performance. Face it?there is no such thing as bad-looking pro forma performance. In designing new products, it's easy to manipulate numbers to fit needs, and pro forma numbers often are not comparable to real world numbers. Vendors want your clients' money. They aren't concerned about the ethics of making investment decisions on the basis of back-tested data.
2. Don't accept short-term track records. They don't tell you anything, except maybe that the fund or manager participated in a great market environment. Insist on actual returns over at least a full market cycle, typically five to seven years. Always look at short-term performance in the context of longer-term trends in the same market or asset class.
3. Insist on third-party verification. By publishing pro forma results from CTAs that could not be independently verified, S&P put its credibility at risk. You will make the same mistake if you don't rely on a third-party firm like Morningstar, Lipper, or BARRA to verify performance records. Some hedge funds will tell you they have never had an independent verification, but don't buy it. Their books are audited every year by CPAs, and it's not hard for CPAs (or reputable administrators) to sign off on performance. In any case, the lack of performance verification is the vendor's problem, not yours.
4. Insist on performance records created by the same manager in the same style. After a fund changes managers or styles, of what value is the old performance record?unless the fund runs on auto-pilot? Don't accept performance records manipulated through mergers of two or more funds into one ("inherited performance"). Especially, avoid performance records that combine theoretical with real results. It is unethical for a professional financial advisor to mix the two to influence clients' thinking.
5. Be skeptical and teach your clients to be the same. In today's industry, you should believe that many performance records contain some element of manipulation. The exceptions will pass all tests described above?namely, they are created by the same managers in the same style over a long period of time, documented by a firm like Morningstar, don't include any inherited performance, and are real (not pro forma) net of all fees and expenses. Tell your clients that performance records should be dismissed or discounted unless you (and they) believe all these tests are met. Focus the investment evaluation process on more objective and verifiable issues, such as the strengths of the investment adviser, the manager's background and experience, and how the style fits each client's needs.

Investing has, in some ways, become a game in which everyone is trained to keep their eye on the ball?performance records. But in so many cases, the ball has been doctored. When your clients swing at it, they are likely to miss hitting their goals. Help them take their eyes off the ball and see the truth.