They also divided the companies into ten groups based on sectors, which ranged dramatically in size, from a high of 84 companies in the financial sector, to only six in telecommunications. 

Companies whose total summary pay was below the median for their sector outperformed those whose pay was above it by 39 percent, the study found.

Related: Public believes executive compensation needs cutting

An investor who put $100 into one of the examined companies in 2006 would have seen his investment grow to $264 among companies in the top 20 percent in terms of executive compensation. Had he put the $100 into a company in the bottom 20 percent, however, his investment would have grown to $367.

The study also focused specifically on equity pay. The conventional wisdom that tying an executive's fortune to the company's stock will motivate better performance does not appear to be supported by any evidence, at least among the largest companies.

Unfortunately, note the study authors, it is difficult for investors to get a firm grasp of how well executives are getting paid. The financial disclosures that companies are required to file include a lot of information that is complex and tough to distill because of the many different forms of compensation, which in recent years has become weighted towards equity and bonuses, rather than salary.

Furthermore, the data is often geared towards short-term performance, rather than the long-term.

Related: 10 CEOs with the highest pay

"Broadly speaking, the sort of cumulative figures that long-term investors would find most helpful are very rarely included," the authors wrote.  "(W)e believe the general lack of such information in individual annual filings may be a likely cause for much of the short-term orientation we encounter in tracking and analyzing U.S. CEO pay practices and reporting."

This study is not the first to suggest that incentive pay is not effective for those at the top of the corporate ladder. While it may be a good way to get better work out of those engaged in menial labor, many experts have argued that tying the outcomes of complex tasks to bonuses is at best futile and at worst counter-productive and dangerous (remember that financial crash thing?).

"Decades of strong evidence make it clear that large performance-related incentives work for routine tasks, but are detrimental when the tasks is not standard and requires creativity," writes the Harvard Business School researchers.