Dunsmore House is one of only two 19th century limestone buildings remaining in Waterloo. The other is the old school at 403 Parker Street, built in 1858. It is believed that Thomas Chadwick, a master stonemason from England, built the Dunsmore House in 1866. The stone was brought from a small quarry located near the present site of Allen Memorial Hospital.

The Chadwick’s originally lived in the house as tenants of the owner, Annetta Ravenscroft. C. O. Farnsworth later owned it before passing to John Dunsmore in an 1873 sale.

Dunsmore was an employee of the Illinois Central Railroad. He eventually became supervisor of the railroad's water works before his death in 1904. His family soon left Waterloo and the house remained vacant until its purchase in 1913 by Fred Michael, the owner of a paint store.

Michael completely remodeled the house, adding a kitchen, a porch, a bath, a sleeping porch, a fireplace, a furnace, electricity, a back stairway and an additional basement area. The house originally had four rooms downstairs and four rooms upstairs, with a central hallway and front and back porches.

Louis Townsend purchased Dunsmore House in 1926. It did not change hands again until 1968, when it was sold to Glen Garrison. The Bicentennial Commission of Black Hawk County purchased the house and adjoining lot in 1974, and a restoration project was begun. There have been several owners of the home, and at present, the city of Waterloo holds the title. Plans have called for the city to present the title to the Waterloo Art Association, which is restoring the building.