Lord Dufferin died from an accidental overdose of morphine and was succeeded by his only son. Helen Lady Dufferin was married secondly after his death to George Hay, Earl of Gifford (1822-1862), the eldest son of George Hay, 8th Marquess of Tweeddale on October 13 1862. He died on December 22 1862, and Lady Dufferin died on June 13 1867.