His short answer was 'Longhorn', the repeatedly delayed and ultimately recast operating system that finally was released as Windows Vista in January 2007.
According to ZDNet, Ballmer said he considered it one of four defining moments of a good part of his CEOship.
During a sit-down interview to tech journalist Mary Jo Foley for Fortune, He talked about his role in key Microsoft events as CEO, from bringing the Xbox to the living room, to taking care of countless anti-trust lawsuits, to taking responsibility for the Longhorn fiasco.
Ballmer said that the most he regretted was not just his CEOship but his whole time here, it's absolutely 'Longhorn becomes Vista.'
That was the single biggest mistake he made, Ballmer said.
Ballmer said he personally takes responsibility for the situation.
With Longhorn, Microsoft started out with the wrong technology approach, the wrong focus by the company's tech leadership and the wrong cadence.
Longhorn kind of fell through the cracks. Ballmer admitted that it was not Bill Gates' thing and it was not former Windows Chief Jim (Allchin's thing but he did not get it.
He said that looking back he realises that it was sort of a focus issue, because they were not focusing on what they needed for engineering cadence, bite-size approach.
Ballmer said he's feeling good about where the company is right now with the Xbox One.