Related: Employee financial health connects to physical health

Advisors need to understand the particular challenges of the plan sponsor's situation. Shouldn't this go without saying? "Advisors need to know our business," said Katie Drayo, Director of Benefits at Cott Corporation, which has gone through several acquisitions. "They sign up for Google Alerts and if they see news about an acquisition, they contact us and say congratulations."

Advisors need to offer "if-then" scenarios and specific examples. "I look to them to bring me the real-life stories – 'hey, we did this with this organization and here's how it worked out,'" said Gloria McCamley, Director of Human Resources, Association of periOperative Registered Nurses (AORN).

And being a good communicator doesn't hurt. "Being able to make all these topics and decisions important and even interesting—that's an extraordinary advisor," said Drayo.

Advisors need to be leaders. "We expect the plan advisor to be an advocate, make decisions that will result in good outcomes. Advocate and lead—that's important to us," Gertzog said.

An advisor will drive strategy, Gertzog said. "I expect someone who is provocative by the questions they ask and the things they bring up that I hadn't thought about."

And he or she has to be able to work with the provider too. "If an advisor and I decide that we're going to do something the provider needs to be involved. Choosing to have a relationship with all three of us together is a big deal," said Drayo. "I don't want to be in the middle of every conversation. I want them to know when to bring me into the conversation."

"Our investment advisor and our provider do work closely. The provider comes to our fiduciary committee meetings. They're talking behind the scenes," McCamley said. "I am working with the advisor from a strategy perspective. I'm trusting that in turn they know our provider's offerings, they're keeping me informed about changes even to their website and to let our employees know about it."

Being involved in discussions is more important to Drayo and her employer. But the advisor has to be able to go off and work independently too. "The advisor and the provider help me get projects done. To know they are going to run a project and yell when there's a question, that works."

She makes sure to build education meetings into the contract with both the provider and the advisor. "If they're built in, they happen. They work with each other, and I get to step out of it and they handle it."

McCamley's organization, AORN, recently changed from a 3(21) to a 3(38) relationship. "Our fees actually went up slightly and I'm willing to pay for this" to be freed up to focus more on certain goals. "We're hiring more millennials and they don't think about retirement, so we want to focus on that, not on investments."

Fees are important, Gertzog said. "They need to be reasonable and market driven. Outliers are a flag to us. I would hope that people are competing on value not cost. It's disappointing to me to wade through a contract and someone says 'I could have done this but we didn't put it in the contract' or nickel and diming."

Accountability is a bit of a squishy area. "It's hard to measure with metrics. But you can reflect back on what you said you'd do, having minutes that actually reflect the meeting and what was said," Gertzog said.

"But if we don't meet our goals, I'm not firing the investment advisor. We're a team. As long as they come with strategies, it's okay," McCamley said