The Octagon was constructed between 1799 and 1801. The Tayloe family of Virginia moved into the house in the fall of 1802.
The Octagon was originally constructed to be a winter residence for the Tayloe family, but they lived in the house year-around from 1818-1855. The Octagon property originally included a number of outbuildings, including a smokehouse, laundry, stables, carriage house, and an ice house (the only surviving outbuilding) The Tayloes were involved in shipbuilding, horse breeding and racing, and owned several iron foundries—they were fairly diversified for a plantation family. The Tayloes owned hundreds of slaves, and had between 12 and 18 who worked at the Octagon.