The Freedom House Museum was once part of the headquarters for the largest domestic slave trading firm in the United States, Franklin and Armfield. Enslaved Africans were brought from the Chesapeake Bay area and forced to the slave markets in Natchez, Mississippi and New Orleans either by foot or ship. The building has a long history. In 1828, it was leased by Isaac Franklin and John Armfield and used as a "Negro Jail" or slave pen for slaves being shipped from Northern Virginia to Louisiana. During the American Civil War the museum and its surrounding sites were used firstly as a military prison for deserters, then as the L'Ouverture Hospital for black soldiers, and finally as the barrack for contraband-slaves who fled the confederate states and sought refuge with Union troops.

The building is currently owned by the Northern Virginia Urban League but the museum is managed by the office of Historic Alexandria. It operates as a reminder to the people of Alexandria of the city's role in historic slavery.

In the basement of the building there is a powerful exhibition which depicts the harsh reality of the domestic slave trade and Alexandria's role in it, through the use of first person narratives from enslaved men and women. These are complemented with a range of contextual text panels, artefacts, images and maps.