SAN CLEMENTE, Calif. - First Pacific Advisors' small-cap to midcap value strategy is off to a decent start this year. Sporting a 15.1% return for the first two quarters of 2003, the strategy was ranked in the 27th percentile by TUCS and the 35th percentile by InvestorForce, according to data supplied by Ron Surz, president of PPCA Inc., a San Clemente performance measurement consultant.
Yet the same separate-account strategy looks mediocre in PIPER, a manager performance database owned by Pensions & Investments, where the so-called SMID approach ranked in the less-than-glamorous 54th percentile. (Stories on the most recent PIPER data can be found on page 15.)
The reason for such disparate rankings stems from a phenomenon known as "classification bias." While the Los Angeles-based manager's SMID strategy was included in midcap value universes by Wilshire Associates Inc.'s Trust Universe Comparison Service, Santa Monica, Calif., and by InvestorForce Inc., Wayne, Pa., according to Mr. Surz, in PIPER, the portfolio was compared to small-cap value managers.
These differences in classification have rankled managers for years.
The problem, Mr. Surz explained, is that very few managers are purists, and they tend to use a blend of styles. "Somebody would rank in the top quartile in Callan (Associates Inc.'s universe) and below median in (Frank) Russell. This is screwy," he said.
Mr. Surz has responded to this and other performance-measurement problems by creating his own synthetic peer-group universes. Popular Index Portfolio Opportunity Distributions - nicknamed PIPODS - creates a synthetic peer group for a manager, based on its benchmark.
After pushing a more sophisticated version for 11 years, Mr. Surz has tweaked his methodology in a way that he hopes will be attractive to pension executives and investment consultants. And he has a handshake deal with InvestorForce to distribute the product on a subscription basis to the online vendor's customers.
"From our perspective, this is just another set of data and information we find may be beneficial to our user base," said Jim Morrissey, InvestorForce's president and chief executive officer.
Some experts praised Mr. Surz's idea, but questioned if this is a Quixote-like pursuit.
"It's 10 times better than what people are using," said Steve Hardy, president of Zephyr Associates Inc., Zephyr Cove, Nev.
Zephyr, a leading style and performance analysis firm, had offered a predecessor to PIPODS. "I always liked the idea intellectually, but there never was a lot of demand for it," Mr. Hardy said.
Added M. Barton Waring, managing director and head of the client advisory group at Barclays Global Investors, San Francisco: "Ron has been showing this for a long time. Unless the big master trust organizations that are doing a lot of performance reporting and the big consultants accept it, it won't be broadly accepted."
Instead of being measured against a universe of actual managers, PIPODS stacks a manager up against a universe of 10,000 hypothetical portfolios using stocks included in a popular index, such as the Standard & Poor's 500 or the Russell 2000 Value, or even a customized benchmark.
The upshot: Managers are measured against the portfolios they could have created, based on their respective benchmarks.
Not only does this get around the classification issue, but it also avoids problems with survivorship bias and composition bias, which stem from weighting certain kinds of managers or lacking enough style-specific managers to form an appropriate universe.
Survivorship bias, in particular, can be an acute problem. As managers drop out of universes over time, their departures tend to hike the median performance while driving down the rankings of the remaining managers. That's because most managers that withdraw from universes have weak returns.
A related problem is that some poor performers are tardy in submitting their data, missing the deadline for the universe's initial distribution.
By creating a synthetic universe of possible portfolio returns, there are no survivorship bias issues, and no arbitrary decisions on how to classify managers or which managers to include in the universe.
PIPODS "really reduces most any kind of bias," said John O'Brien, executive director of the Masters in Financial Engineering Program at the University of California at Berkeley's Haas School of Business. Mr. O'Brien, with Mr. Surz's aid, developed a forerunner of PIPODS in the 1980s when they worked at A.G. Becker Funds Evaluation Group (now SEI Investments Inc.)
What's more, instead of waiting nearly a month at the end of a quarter to get comparative results, data from Mr. Surz's synthetic universe is available a couple of days after the end of each month.
PIPODS also offers pension executives the holy grail of investing: identifying managers that show skill. While it could take decades to show that a manager has actually shown skill instead of luck in beating a conventional benchmark, Mr. Surz said a manager ranking in the top decile in his methodology has a 90% probability that skill, and not luck, was responsible for his success.
The other side of the coin, though, is that the manager could be investing well outside of his benchmark. Either way, that superior performance clearly warrants closer scrutiny.
The PIPODS challenge
To put PIPODS to the test, Mr. Surz generated rankings for both six-month and one-year periods ended June 30 for five strategies with top 10 performance in PIPER over the last 10 years.
Those strategies are:
c First Pacific's small-cap to midcap value strategy, benchmarked against the Russell 2500 Value index;
c Artisan Partners' Mid-Cap Growth strategy, tied to the Russell Mid-Growth index;
c Janus Capital Management LLC's large-cap growth strategy, measured against the Russell 1000 Growth index;
c Waddell & Reed Asset Management Group's small-cap growth equity strategy, benchmarked to the Russell 2000 Growth index; and
c Legg Mason Capital Management Inc.'s celebrated value equity strategy, measured against the S&P 500 index.
The PIPODS rankings then were compared with rankings for the same separate-account composites in peer universes created by InvestorForce, PIPER and TUCS, using data supplied by Mr. Surz and PIPER.
The results are detailed in the accompanying table: Artisan looks like it has been in for tough sledding in PIPODS for both periods, but even its 85th percentile ranking for the six months is not statistically significant, failing to reach the magic 90% confidence level, Mr. Surz said. (While the number shows the manager underperformed its potential universe, it's not clear whether luck or skill is responsible for the poor performance. Statisticians use 90% as a key test.)
First Pacific, Janus and Waddell & Reed also come off worse against their respective hypothetical universes than against their peers. Their weaker performance in the second quarter raises the question as to how well the managers have been faring in the stock market bounce-back this year, he added.
The sole shining light remains the legendary Bill Miller at Legg Mason, whose value approach is ranked in the first decile in PIPODS, InvestorForce and PIPER for the one-year period ended June 30.
Tough road ahead
Mr. Surz still has a tough row to hoe. Since 1992, he has hawked a more sophisticated version of PIPODS. The main difference was that instead of generating a cyber-universe of returns off a standard benchmark, Mr. Surz had created his own style indexes.
But the idiosyncratic indexes, on top of synthetic peer groups, were a bridge too far for most potential users. "He was fighting two wars," BGI's Mr. Waring said.
"Maybe now, with InvestorForce behind it, (PIPODS) will get more of the recognition that I believe it deserves, but we'll have to see," Mr. O'Brien said.
For his part, Mr. Surz is undaunted. Citing Malcolm Gladwell's "The Tipping Point," he notes it has taken many years for some major financial ideas - from Modern Portfolio Theory to Black-Scholes' option-pricing model - to catch on. One large, visible pension fund or hundreds of less visible funds and consultants could make the difference, he said.