In 1171, Henry married a second time, this time to Agnes, daughter of Henry of Guelders and Agnes of Arnstein. Heirless still, he repudiated her in 1184, but fell seriously ill and rejoined her. In September 1186, a girl, Ermesinde, was born to them. This birth called into question the plan of succession, as Henry considered his promise to Baldwin null. Henry, then 76 years old, pledged his daughter in marriage to Henry II of Champagne. Baldwin still claimed his inheritance. It was then decided that Baldwin would inherit Namur, Ermesinde Durbuy and La Roche, and Luxembourg (fief masculin) would revert to the Empire.
The fiefs were dispensed in 1189 and after the planned marriage between Ermesinde and the count of Champagne was cancelled Henry bethrothed her instead to Theobald I of Bar. He entered into a war with Henry of Limburg and was defeated on 1 August1194 at Neuville on Mehaigne. He died two years later in Echternach.