Orford was once a small fishing village situated to the north of the castle. From the time when the Castle was built between 1165-72 and on into the 13th century Orford gradually grew into a town of importance. There were weekly markets in fish, wool and other merchandise, and merchants from the continent would visit the town.
The castle was built for Henry 2nd as a defence against local baronial interests wishing to usurp the crown in East Anglia. Only the keep remains in an exceptional state of preservation and gives a real sense of how life was lived in the 12th century. Recent research by Dr T A Hislop of the University of East Anglia has established it as one of the most remarkable English castles.
The total cost of building was £1407-9s-2d. This sum is recorded in documents known as the Pipe Rolls dating from Norman days. These were financial statements made yearly to the King's Exchequer in London by the Sheriffs of the various counties.
The keep is about 63 feet high from the floor of the basement to the floor of the battlements. Three turrets rise from there about another 20 feet. The walls are very thick and constructed of mainly London Clay (septaria) which was obtainable in the neighbourhood. In the basement is a well 4 feet 3 inches in diameter and 37 feet deep with holes in the masonry for lodging planks to enable the well to be cleaned.
In 1262 Henry 3rd granted the Castle to his first born, Edward, on his 23rd birthday. It subsequently changed hands many times until it passed with Sudbourne Hall to the Lord of the Manor of Sudbourne. One owner was Sir Michael Stanhope, Groom of the Chamber to Elizabeth and James 1st. Later owners were successive Marquesses of Hertford and Sir Richard Wallace founder of the Wallace Art Collection. By the generosity of Lord Woodbridge, then Sir Arthur Churchman, MP for the Woodbridge Division the castle was conveyed to "the Town Trustees of Orford for the Town and Nation to be a joy forever".
Eventually, after the Town Trustees had spent considerable sums of money on repairs and conservation, it was realised that the expenditure was beyond the towns means and it was assigned by deed of gift to the Ministry of Public Buildings and Works. In hindsight perhaps it would have been for the benefit of the village if it had been loaned rather than given away. It is now administered by English Heritage.