The castle was built around 1800 by the Lord of Walzin, a district of Dinant. At the time it was known as the ‘Château de Lesse’ and consisted of a main building and tower with two floors. Lady Marie-Anne Damsel de Colnet was the first person to live in the castle.
Then it came into the hands of Ferdinand Louis de Villers-Masbourg, who passed it on to his three children in 1844.
In 1868 the Villers-Masbourg family sold the castle to Marguerite Offerman, wife of Frédéric Guillaume Brugmann. This Brussels banker had already become the owner of the famous Walzin Castle, which towers out on a rock above the river Lesse, in 1850.
Castel Pont-à-Lesse then fell into very poor condition, so son Alfred Maurice Brugmann had it fully rebuilt over many years.
In 1914 the castle was again fully renovated by grandson Frédéric Brugmann de Walzin, who added a third floor. He renamed the castle ‘Kasteel van Pont-à-Lesse’.
The castle was then sold in 1948 and tuned into a holiday centre. Since then the Castle has been a distinguished three star hotel.