The New York Times reported on Ayers' trepidation Saturday, while also naming Treasury Secretary Steven Mnuchin, budget director Mick Mulvaney and USA trade rep Robert Lighthizer as possible candidates.
"The question is going to be whether a new chief of staff can have the same kind of gravitas that John [Kelly] did, which is the ability to stop the wrong decision from being made", said Leon Panetta, who was chief of staff for former president Bill Clinton from 1994 to 1997, and later became Central Intelligence Agency director and secretary of defence in the Obama administration.
Ayers, Axios reported, will run Trump's outside group, America First, after leaving the administration.
As President Donald Trump ponders picking a new chief of staff, more names are emerging as possible contenders for the job.
Nick Ayers, who was widely speculated to be the next White House chief of staff, will not be stepping into John Kelly's role, ABC News has confirmed.
Trump said Saturday that he expected to announce a replacement for Kelly in a day or two.
Kelly replaced former White House chief of staff Reince Priebus in July 2017.
Problematic for Trump, already, three candidates for the job said they would prefer not to take the job, Politico reported. Trump increasingly bypassed Kelly to talk to old friends and outside advisers, tweet or make tactical moves against Kelly's advice.
-Mick Mulvaney, director of the Office of Management and Budget.
According to White House officials, Kelly's influence waned over the last few months with the president undermining the authority that he had established in getting the White House in order.
At the time, Trump said he believed Kelly would "do a spectacular job", and lauded his time at the Department of Homeland Security (DHS), where he oversaw a reduction in the number of illegal border crossings and implemented Trump's travel ban on people from terror-prone countries.
Ayers and Trump had discussed the job for months.
Kelly, whose last day on the job is set to be January 2, had been credited with imposing order on a chaotic West Wing after his arrival in June 2017 from his post as homeland security secretary.