"Fees in the asset management industry are coming under increasing scrutiny, and this trend has driven investment dollars into lower-cost funds, particularly index funds," said Patricia Oey, a senior research analyst at Morningstar.

Morningstar separates funds into five quintiles by cost. Funds in the lowest-cost quintile saw $303 billion of inflows last year, and $1.7 trillion of inflows over the past five years. Passive funds and indexed funds such as ETFs accounted for 75 percent of those flows.

Read: More advisors say DOL rule could help them but hurt investors

Funds in the four other quintiles saw $260 billion in outflows in 2015, and $372 billion in outflows over the past five years.

Last year, the average expense ratio for passively managed funds was 18 basis points, compared to the 78 basis-point average for active funds.

Notwithstanding that discrepancy, actively managed funds still dominate the market, accounting for 70 percent of all funds.

But that gap is closing; the nearly 30-percent market share of passive funds is up from 14 percent in 2005. There are eight actively managed funds for every passive fund in the market.

Among all asset classes, the largest difference between active and passive fund fees is in U.S. equity. The average U.S. equity actively managed fund costs investors 79 basis points, compared to the 12 basis-point average for passive U.S. equity funds.

Passive U.S. equity funds saw $471 billion of inflows over the past five years, compared to $572 billion of outflows with active funds.

Taxable bond funds, an asset class that historically has used active strategies, are increasingly deploying passive management, as more target-date funds use index funds for their bond allocation.

Vanguard's Total Bond Market II fund is the largest passive fund in the taxable bond category, accounting for 22 percent of assets in all passively managed taxable bond funds. Morningstar says that about half of the $96 billion in the Total Bond Market II fund are held within Vanguard's target-date series.

Over the past five years, Vanguard funds experienced the largest price decline among major fund families. The average Vanguard fund fee fell from 18 basis points to 12 basis points in that period.

That no doubt helps explain Vanguard's growing market share, which increased from 15 percent to 20 percent over the five-year period.

Fidelity funds accounted for 9 percent market share in 2015, equal to its market share in 2010, and American Funds accounted for 8 percent of the market, down from 11 percent in 2010.

SPDR State Street funds averaged 19 basis points, the second lowest behind Vanguard, and Dimensional Fund Advisors funds averaged 36 basis points.

Of the ten fund families Morningstar examined, four experienced outflows over the five-year period:

• Fidelity at $76 billion

• American Funds at $155 billion

• Franklin Templeton at $21 billion

• Pimco at $168 billion

Morningstar's report says the Department of Labor's recently released fiduciary rule is likely to result in more fee scrutiny and lower investment expenses for retirement savers.