But with stocks having more than doubled in recent years and sneaking toward a five-year high, those who are no longer playing the game run the risk of missing out.

"This is the most uncelebrated bull market in history," said Tony Ferreira, Cogent Research managing director. "In the old days, people would be jumping on the bandwagon, but nobody's chasing equity performance this time. Many people are still scared to wade back into the water."

Those with major investments are reaping the returns, but investors with less than $100,000 in the market are not seeing those same gains. Cerulli reserarch shows that the $100,000-and-under crowd had, on average, just $17,975 left in investments, down 9 percent from $19,732 at the end of 2007. Those with half a million to $2 million in assets have seen a 7 percent growth in their savings and investments. 

Investors didn't pull out all their stock holdings, but ownership of equities has dropped significantly. Equity ownership in stocks, mutual funds, variable annuities and ETFs reached 53 percent in 2001 and is down to 46.4 percent in 2011.

Investors have yanked $236 billion out of U.S. equity mutal funds since 2007, including $53 billion since last October. Yet since that time, Reuters notes, Standard & Poor's benchmark has gained 28 percent and the Dow Industrials are up 24 percent.

Fidelity's numbers show that investors who retained holdings in equities had an average balance $45,000 more than those who liquidated their equities in 2008 or 2009.