The name, the Garden District was selected by the Toronto East Downtown Residents Association (TEDRA) in recognition of Allan Gardens, an indoor botanical garden located nearby at the intersection of Carlton and Jarvis Streets. Part of the neighbourhood is within the official City of Toronto neighbourhood of Moss Park.

The neighbourhood is bordered by Carlton Street to the north, Yonge Street to the west, Sherbourne Street to the east and Queen Street to the south. This area includes the southern part of Toronto’s Gay Village and heritage sites such as the Mackenzie House Museum, Gallery Arcturus and the Merchandise Building. 

The western portion, from Yonge Street to Jarvis Street, was first subdivided at the time Toronto was incorporated as a city in 1834 from the McGill Estate (the owner John McGill died that year). This area quickly filled with overflow from central Toronto and includes a number of early, mostly non-government (especially religious) institutions, places of entertainment such as Massey Hall, the Elgin and Winter Garden Theatres, the Canon Theatre (formerly Pantages Theatre) and Maple Leaf Gardens as well as the Ryerson Model School (now a part of Ryerson University).

The eastern portion from Jarvis Street to Sherbourne was largely subdivided later in the 1850s from the Jarvis Estate (Hazelburn) and the Allan Estate (Moss Park), with north-south streets based on the street grid of the original Town of York. This area is largely residential with two parks (Moss Park and Allan Gardens) and a large number of local churches. It was originally a very exclusive district and declined in time as the housing stock aged and smaller lots for workers were built. This area has seen a large replacement of older homes with residential, institutional and commercial development. Located in the eastern portion of the Garden District, Walnut Hall were four Georgian-styled terraced homes. On May 18, 2007, local uproar arose over its demolition. Constructed in 1856 and designated as a heritage site in 1997, the building was nonetheless neglected and began to collapse.

The Garden District includes a mix of housing, from million-dollar condos, renovated Victorian villas, and Edwardian row houses to apartment co-operatives, subsidized housing units, and many hostels and shelters. George Street, one block east of Jarvis has a blighted area around the Seaton House shelter. The City of Toronto government is planning to redevelop the site incorporating heritage homes now boarded up.