In an e-mailed statement, Puzder said he had hoped to "put America’s workers and businesses back on a path to sustainable prosperity" and bring "new thinking" to the job.

Trump has already lost one senior member of his administration. He dismissed Mike Flynn as national security adviser on Monday because the administration said he may have misled the president and vice president about his communications with a Russian envoy.

Civil rights commission

Kirsanow, now a member of the U.S. Commission on Civil Rights, met with Trump in November on labor issues. He would be the second African-American member of Trump’s cabinet. As a private attorney, he represented the National Association of Manufacturers in a successful challenge to an Obama-era NLRB rule requiring employers to display more information about employees’ union rights.

Acosta served as assistant attorney general for the Civil Rights Division under President George W. Bush and was U.S. attorney for the Southern District of Florida. Either Acosta or Guzman would be the first Hispanic nominee for Trump’s cabinet.

Guzman, who teaches labor relations at Michigan State, was a member of the Trump campaign’s National Hispanic Advisory Council. Trump summoned him for a meeting on Jan. 3, when he was working through his original cabinet picks.

Templeton, a lawyer known for anti-union work, served as South Carolina’s labor secretary and later led the state health department, appointed by Governor Nikki Haley. She said last fall she intended to declare her candidacy for governor in South Carolina.

Trump met Wednesday night with Acosta and Guzman; he meets Friday with Templeton in South Carolina, the White House official said.

Related: Trump's pick to head OMB staunch opponent of Labor fiduciary rule

At least six Republicans had said they weren’t ready to back Puzder and said they were waiting for his confirmation hearing. CNN reported earlier Wednesday that top Senate Republicans urged the White House to withdraw Puzder’s nomination because at least four, and as many as 12, GOP senators planned to vote against confirmation.

In the 52-48 Senate, three Republican defections would have doomed Puzder if all 48 Democrats voted to deny him.

Trump’s Labor Pick Loves Burgers, Bikinis, and Free Markets

Puzder is head of the fast-food conglomerate CKE Restaurants Inc. that includes Carl’s Jr. and Hardee’s. His withdrawal comes after a week that saw party-line confirmations in the Senate and a tie-breaking vote cast by Vice President Mike Pence to confirm Education Secretary Betsy DeVos. That 51-50 tally was the first time a vice president ever broke a tie on a cabinet nomination.

Puzder acknowledged early this month that he employed an undocumented immigrant as a housekeeper until learning of the worker’s immigration status. Senate Majority Leader Mitch McConnell, a Kentucky Republican, had said last week that allegations of hiring undocumented workers have sunk other cabinet nominees, but not always.

Senator Tim Scott, a Republican from South Carolina, said in a Twitter posting after Puzder announced his withdrawal that he initially was "impressed" by Puzder’s vision for the Labor Department, but in recent recent days he told Republican leaders that he wouldn’t support the nominee.

"As revelations regarding paying employees in cash, illegal immigration and comments regarding some of the American workforce came to light, I developed serious concerns," Scott said in the posting. "I look forward to meeting with a new nominee."

Opponents also seized on multiple news reports that Puzder had been accused by Lisa Fierstein, his first wife, of domestic abuse. Puzder denied the allegations. Fierstein, in a Jan. 18 letter to lawmakers, said that she had made such allegations against Puzder but later withdrew them, saying “Andy is not and was not abusive or violent.”

Oprah Winfrey Show

Republican Senator Susan Collins of Maine -- one of those who expressed doubts about Puzder earlier this week -- said she had viewed an episode of Oprah Winfrey’s talk show dealing with domestic violence in which Fierstein appeared. "I am reviewing the other information that has come to light and I’m sure all of this has been explored thoroughly," Collins said at the time.

Puzder had agreed to divest an ownership stake worth as much as $50 million in his closely held fast-food company if he was confirmed as labor secretary, according to his federal financial disclosure and ethics agreement.

Democrats sharply criticized Puzder’s nomination, as Minority Leader Chuck Schumer of New York called his withdrawal a "victory" for workers.

"Puzder should never have even been nominated to lead the Labor Department, and Senate Republicans clearly recognized this too," Schumer said in a statement. "The fact that someone so anti-labor was even nominated shows how far President Trump is from where he campaigned."

Puzder opposes efforts to raise the minimum wage. He also has written that legal immigration was an asset to the nation and that it would be unworkable to deport the 11 million undocumented people in the U.S., as Trump promised repeatedly during his campaign.

But he also said Trump’s proposals to build a border wall, stiffen enforcement and defund cities that provide sanctuary to undocumented immigrants were reasonable.

Related: Doomsayers got it wrong on higher minimum wages

Karl Frisch, executive director for Allied Progress, a labor group that fought Puzder’s nomination, said, "Working Americans have a simple message to President Trump: we’re watching you and we’ll hold you accountable.”

Copyright 2018 Bloomberg. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.