Button walked away from a full-time Formula 1 race-seat at the end of last season, opting instead to take on an ambassadorial role with McLaren.
The Brit, though, could have potentially stayed on if he wanted to having spoken with Williams about a race-seat.
In the end, Button decided that it was time to call it quits.
“I’ve almost rejoined Williams I think three times, or four times,” the 2009 World Champion told CNN’s The Circuit.
“I spoke to Williams about racing this year, a couple of times.
“I spoke to them last year and I spoke to them this year as well.
“It’s a great team but, for me it was time.”
Asked if he would throw his hat into the ring if Felipe Massa retired at the end of this season, Button simply replied: “No.”