Although a popular and articulate MP, he was hurt by the tradition of appointing no more than one Canadian Cabinet minister from Newfoundland at a time, and by his reputation as a maverick who said what he thought rather than what the party leadership would like him to say. Fred Mifflin's and Brian Tobin's appointments to cabinet following the 1993 election meant Baker had to remain on the backbench. Tobin's resignation from the cabinet to become Premier of Newfoundland and Labrador opened the way for Baker to be appointed as Minister of Veterans Affairs. But when Prime Minister Chrétien lured Tobin back to Ottawa for the 2000 election, Baker was removed from Cabinet. He was appointed to the Senate in 2002.
Baker lives in Gander, Newfoundland and Labrador, with his wife Averill Baker.
The treatment of Baker echoed that of James McGrath during the Mulroney years. A popular and talented MP, McGrath would have likely been in cabinet had a seat not been given to John Crosbie.