In 1839, renovation work began on what would become the current neoclassical building belonging to the second branch of the Olives family. Archduke Luis Salvador of Austria, from the imperial Habsburg dynasty, described it in his Die Balearen (1869-1891) as "without a doubt the most beautiful mansion on the island."

The palace rooms have witnessed visits by kings and other distinguished figures, its seizure by the Republican army during the Spanish Civil War (1936-1939), family celebrations and social events hosted by the various counts who over time have been invested with important public offices, in the former Universitat General de Menorca, the mayor's office of Ciutadella, and the Asamblea Nacional Consultiva during the reign of Alfonso XIII, among others, and honours from the Royal House and Heritage of the Crown of Spain.

The old mansion, the ancestral home of the Torre-saura family for 177 years (1818-1995), jealously guards within its walls an important part of the history of the city and of Menorca, as well as the atmosphere and memory of the unfortunate fate of one of the main lineages of the island.