The old fortress near the centre of town is the remains of the castle of Brecknock, built by Bernard de Neufmarche from 1093, within 25 years of the Norman conquest. As well as a military headquarters it was also the administrative centre for the Lordship of Brecknock following the defeat of Rhys ap Tewdwr.
The ruins feature the earliest style of Norman architecture. The castle was later enlarged by the De Braose family, Norman overlords who succeeded Bernard de Neufmarche and ruled the region for nearly a century. They were followed by the de Bohuns who reigned for a further 130 years. Although the castle came under siege by Prince Llewelyn in 1217 and again in 1233, it was never taken by force. Henry IV possessed it in 1404. The fortress was successfully defended against the Welsh patriot, Owain Glyndwr, by 100 men at arms and 300 Welsh archers. They were reputedly paid sixpence a day for their valour. Later, Henry, Duke of Buckingham, became Lord of Brecknock, and his son, who was eventually executed in 1521 on the orders of Henry VIII, enlarged and improved the castle. It later passed to the Crown and, eventually, both town walls and castle were partially torn down to avoid a siege by Parliamentarians during the Civil War. The castle property passed to the wealthy family of Baron Morgan of Tredegar, who built most of the Regency house. Part of the castle wall is now within Ely Tower, the residence of the Bishop. It was here that a scheme to overthrow Richard III and to prepare the accession of Henry VII was plotted. It is, however, still possible to get a fine view of the ancient castle town and walls from the gardens of The Castle of Brecon Hotel.