Bailey married, firstly, Maria, daughter of Joseph Latham, in 1810. In about 1826 he bought Glanusk Park and had a mansion house built there. After his first wife's death in 1827 he married, secondly, Mary Anne, daughter of John Thomas Henry Hopper, in 1830. He died in November 1858, aged 75, and was succeeded in his title by his grandson Joseph Russell Bailey, who in 1899 was elevated to the peerage as Baron Glanusk. Lady Bailey died in 1874.