The house was owned by the Duke of Buckingham and because he held manorial courts here, it became known as the Manor House.