A chapel of Holy Trinity was erected next to the southern part of the buildings, which was painted by the hand of an unknown master in 1471.
The south-western and north-western towers were demolished after there were no longer war fears. This happened in the second decade of the 18th century, when the parish church was expanding to the west (1713) and the construction of its baroque façade (1717) was taking place. The north-eastern tower was demolished in 1943 when Žminj was bombarded during World War II. Today, only one tower has been preserved along with the south wing of the old citadel, which is incorporated into the parish church. The remaining tower has been enriched by a balcony immediately after the war dangers ceased to exist (probably after the Uskok war in 1618). The walls of the citadel were preserved in its northern part and were repaired in the 15th century.
Žminj represented an important fortress, primarily because it stood as an Austrian fort opposite the Venetian one in Svetvinčenat. In preparation for the conflict with Venice, Austrian archduke gathered around 200 infantry soldiers. During the war, from 1615 to 1618, Žminj’s infantry often attacked the Venetians, destroying their crops, especially the hay which was transported from the port of Fažana to Svetvinčenat. Because of the lack of hay, Venetian cavalry was almost powerless. However, as often happens, a traitor secretly led the Venetians into the castle. For his service he received a lifelong salary of six ducats per month, but fearing for his life he moved to Crete.