Harbord Village lies just to the west of the University of Toronto, with its most commonly accepted borders being Bloor Street on the north, Spadina Avenue on the east, College Street to the south, and Bathurst Street to the west. The area was previously known as Sussex-Ulster, after two of the major east west streets in the area. In 2000, the residents' association decided to rename itself and the area Harbord Village, after the main street running through the middle of the community. The street's name origin is unclear but could be named for abolitionist Edward Harbord, 3rd Baron Suffield. 

The area was primarily opened up for development around 1870, with the initial construction of modest working class houses. Somewhat grander middle class houses began to appear in the late 1880s and 1890s, after a local school had been built and the city had completed paving the streets and installing water and sewer lines. These predominantly Bay and Gable semi-detached houses dominate the current streetscape. Only one block on Brunswick Ave., which was not developed until the 1890s, contains the kind of large houses common to the Annex neighbourhood to the North. In 2005, the Harbord Village Heritage Conservation District was established in the area of Lower Brunswick Avenue (below Ulster St.) as well as Willcocks Street west of Spadina, and then was expanded to include Robert Street and Sussex Avenue. The Harbord Village Residents' Association hopes the entire area will become a heritage conservation district.