(FORTUNE Magazine) – It's been almost 30 years since Vanguard founder John Bogle started preaching the benefits of indexing for ordinary folks. And reams of data long ago convinced many investors that, after trading costs and taxes, very few active stock pickers beat the market over the long haul. That explains the $1 trillion invested in funds indexed to the S&P 500 alone. Billions more are following other indexes. But investors who apply the strategy to small-cap stocks need to check their returns. It turns out that not all indexes are created equal.

Take, for instance, the Russell 2000, the most popular small-cap index, which has $42 billion riding on it. According to Standard & Poor's, the Russell trailed S&P's Small Cap 600 index by a startling annual margin of 3.24% over the past seven years. Morgan Stanley claims that its recently launched MSCI U.S. Small Cap 1750 would have beaten the Russell by 3.08% annually over the same period.

The diverging performance is drawing the attention of big index investors. In May 2003, Vanguard migrated many of its funds to MSCI indexes, including all three of its small-cap index funds, which were previously linked to either the Russell 2000 or the S&P 600. Vanguard chief investment officer Gus Sauter says while choosing among benchmarks is not all that important in the large-cap arena, "it's definitely important in the small-cap segment."

The discrepancy between indexes is partly a result of how they were designed. Russell launched its 1000, 2000, and 3000 indexes in the early 1980s simply as benchmarks that active investors could use to gauge their own returns. "Passive investing," says Lori Richards of Russell, "was never their intention." By contrast, many indexes created more recently, such as those from Morgan Stanley, offer some advantages for passive investors.

How so? For one thing, MSCI allows some flexibility on the size of stocks in each index to cut down on turnover. The Russell 2000 reconstitutes its membership once a year, in July, according to strict market-cap guidelines. It also includes a greater number of smaller companies, which tend to be more volatile. That leads to more turnover and thus increased trading costs for funds that mirror the index. Also, because Russell pre-announces its annual changes to the index each June, the index is subject to widespread arbitrage. Big traders buy shares that will be added and sell those that are to be dumped, effectively penalizing passive investors who wait for the changes to become official before trading. According to a study released in May by Virginia Tech finance professor Vijay Singal and two colleagues, the early trading costs investors in Russell 2000 funds as much as 1.84% of potential annual return. (Russell disputes the study's figures but recognizes the potential for an arbitrage effect.) Because the MSCI makes fewer changes, it's less vulnerable to such arbitrage.

The lesson? When it comes to choosing your index, don't be too passive.