In a DAR booklet of 1901 describing "Old Houses of Norwalk," it is stated that: "The Rector of Saint Paul's at the time of the Revolution was Rev. Jeremiah Leaming, D.D., who had continued to pray for King George notwithstanding the remonstrances of many of his parishioners. He was therefore highly unpopular; and, on one occasion Dr. Leaming and some of his Tory friends were locked up in a barn on the premises now occupied (1901) by Mr. C. A. Tucker. It is therefore not surprising that he decided to leave Norwalk with the British troops after the fire."
In the National Register of the Society of Sons of the American Revolution, Principal Events of the American Revolution, it is stated that Charles Arthur Tucker was the great-grandson of Joseph Tucker, who served in capturing Tories. So -- it seems as though a barn on the Tucker property would have been a likely place for confining Tories.