Consistency. Brinker International—which owns Chili's Grill & Bar and Maggiano's Little Italy restaurant chains—has 60,000 employees and more than 1,000 locations. So people like Kent are looking for brokers who can be consistent with "lots of locations and lots of employees."

It's not about the frills. Parmly said he's not just looking for a broker to take him out to dinner at a nice restaurant. "The people in the background really make the difference—we have a problem and the problem gets solved…that's what's important."

Perfect timing. Kent said she gets anywhere from 25 to 50 calls a week from brokers looking for business—and that can deter her from getting back to them. "If I don't respond it's not that I'm not interested in it, it's because of the volume," she said. But sometimes a broker might nail it on the head by doing a little research about the company first and really pinpointing a product that the employer needs.

An advisor. "I love when brokers come in as an advisor," Kent said. She said she looks for a strategic partner—someone who will help with budgeting and long-term planning, who keeps a "finger on the pulse" and who is technology driven. "I'm looking for someone pretty cutting edge. I have a young demographic, so ahead of the curve is what I'm looking for."

Colleague recommendations. Recommendations from colleagues who are at the same level in the same industry are the most powerful tool for selecting brokers or products, Parmly said. "When we talk to our colleagues about a broker or a product, we trust what our colleagues say more than we do a broker. I don't make money from my friend."