Perhaps ironically, Americans "see themselves as savers and they worry about capital preservation," said Robert Stammers, director of education for the CFA Institute, about the study. 

"They don't take the risk necessary to achieve the returns that they need to fulfill their long-term investment goals," he added. 

Predictably, those with incomes under $30,000 a year were most likely to cite not having enough money to invest as a reason for not being in stocks. 

Only 26 percent of adults under the age of 30 said they owned a stock in the survey. 

The continuation of that trend is likely to catch up to millennials, who will miss out on returns over time, when even small contributions to 401(k) plans can go a long way. 

"Stocks aren't only for the rich," said Claes Bell, an analyst for Bankrate.com. "The key is to have an investment plan in place that aligns your investments with your risk tolerance and goals." 

Of the Americans surveyed who reported not owning stocks, 21 percent claimed the reason was they didn't "know about stocks." 

Millennials were twice as likely as other age groups to stay out of stocks for a lack of knowledge, suggesting a huge need for investor education in younger segments of the workforce. 

Nine percent of those that don't own stocks said they didn't trust stockbrokers. Pre-retirees, those ages 50 to 64, were most likely blame mistrust of brokers and advisors for not investing in stocks. 

Also read: More 401(k) millionaires than ever

Stammers said the bad taste left from the last financial crisis has yet to leave many older investors, and that bad news tends to be amplified by the media. 

"It's funny because most of the people who stayed in the market actually did OK," he said.

"You hear a lot of news about how bad it was, not a lot of news about what happened to people who actually stayed invested."