The poll of more than 2,000 adults, conducted by Harris Poll on behalf of Accolade, an on-demand health care concierge for employers, health plans, health systems, and employees, found that 32 percent of Americans say they are uncomfortable with their personal knowledge of medical benefits and health care.

Read: More physicians bypassing insurers

In contrast, slightly fewer (25 percent) said they felt the same about searching for housing and far fewer (15 percent) said they lacked the knowledge necessary to make a good choice on a car.

What annoys consumers the most?

According to the poll, more than half are bothered by having to coordinate different aspects of benefits and health care, half are flustered in their attempts to understand what their benefits provide and 41 percent find it frustrating to coordinate care across different providers, such as primary care physicians, specialists and hospitals.

Read: What's ahead for PPACA in 2016?

Another point in the survey that Accolade sought to highlight: 80 percent would like to have a go-to person or resource to help them with all of their health care decisions, from choosing benefits to deciding on providers and treatments.

"They need more help understanding and managing their healthcare process, and aren't taking full advantage of the various programs and technologies available to them," said Robert Cavanaugh, president of field operations at Accolade, said in a press release. "By understanding people as individuals with a unique set of financial, medical and life impacts, employers and health plans have a tremendous opportunity to help people become more knowledgeable, confident and empowered to make the best healthcare decisions and achieve and maintain optimal health."