Palm Gardens, at 310 W. 52nd St. in Manhattan, was originally known as Palm Garden and had been a two-story dance hall as far back as 1922 (see Certificate of Occupancy here). The 1922 Certificate of Occupancy indicates both floors were ballrooms, with room for 500 persons on the first floor and 200 persons on the second. Interestingly, the 1923 and 1924 COOs indicated the first floor was a theater (theater and ballroom in 1924) holding 280 people and the second floor was a theater (and ballroom in 1924) holding 20 people. According to the 1927 COO the building once again had room for 500 on the first floor and 200 on the second. I'm not sure what to make of the reported capacity in '23 and '24. Perhaps there were structural issues that caused the city to limit the number of people, or perhaps for a few years in the 20s there were more restrictive occupancy rules.
It had been known as Palm Garden(s) since at least 1936. Palm Gardens was BIG - at some point it could hold over 1000 people.
The hall was not only a ballroom but frequently served as a venue for musicians' union meetings and speeches as noted in many issues of Billboard.
Palm Garden in 1939:
A view of the interior (sort of), 1944:
Photo from the US Coast Guard's history of the destroyer escort USS Joyce (that's the crew and their dates in the photo). Pdf at www.uscg.mil/history/webcutters/Joyce.pdf
Some of the notable people who performed/spoke at Palm Gardens:
1928 - Voodoo, an opera by the African-American Harry Lawrence Freeman opened. It was the first opera by an African-American to be presented on Broadway.
Nov. 1, 1955 - The 29-year old "rebel leader" Fidel Castro spoke to an audience of 800 Cuban exiles and "a few observers under cover", a reference to the FBI and CIA who were keeping tabs on Castro. Castro was on
a speaking tour of the US to call on exiles to help finance a revolution. Apparently at Palm Gardens that night money poured in from enthusiastic Cubans. 
Oct. 27, 1957 - Coltrane performed with Monk at an evening concert at Palm Gardens. He also performed an afternoon concert there on the same day with Donald Byrd, Art Taylor, Tommy Potter and Red Garland.
April 8, 1964 and January 7,1965. Malcolm X gave speeches at Palm Gardens for the Militant Labor Forum, "The Black Revolution" in '64 and "Prospects for Freedom in 1965" in '65. Interestingly, the audiences for
these speeches were primarily white. The entire texts of these two speeches are available in the book "Malcolm X speaks: selected speeches and statements" by Malcolm X and Georger Breitman. You can read
parts of this book, including the speeches, at Google Books. 
Dec. 22-24, 1967. The Grateful Dead play the Palm Gardens! See the Village Voice ad here
Various dates in 1967: Palm Gardens was home to The Group Image. I'll say no more except to direct you to the awesome blog It's All the Streets You Crossed Not So Long Ago.
On March 28, 1968, shortly after the Dead played there, the Cheetah, the well-known club at 53rd and Broadway, moved into the Palm Gardens building. At some point the Cheetah became a popular Latin-American
dance club that helped popularize Salsa to mainstream America. The film "Our Latin Thing", recently restored and available at fania.com, documents an August 21, 1971 concert by the Fania All-Stars, a continuously revolving line-up of entertainers that recorded on the Fania Records label. Fania Records was the leading salsa record company of the time. As stated in an August 2011 NY Times article, "In the history of salsa music and Fania Records, which for many years were all but synonymous, “Our Latin Thing” and the Cheetah show occupy a singular position. It took another Fania All-Stars concert, this time for a crowd of more than 45,000 people at Yankee Stadium in 1973, to alert mainstream English-speaking America to the vast commercial potential of the Latin music market, but it was the Cheetah performance that may have been the ensemble’s artistic pinnacle." Read the article here. There are also videos titled "Live at the Cheetah nightclub 1973" here.
The Cheetah remained at 510 W. 52nd until 1973 but in 1974 the building was purchased by Studio Instrument Rentals and became their first east coast office. SIR is
a music entertainment company that continues to thrive nationwide. SIR sold the building to investors in 2004. It and neighboring buildings were torn down in 2004 and replaced by a 40-story luxury (million dollar +)
condo tower. Bleah.
The Palm Garden building in 1974:
Photo from Studio Instrument Rental history
310 W. 52nd St. today.
As an aside, from 1895 to 1928 there was a Proctor’s Pleasure Palace Palm Gardens nightclub on E. 58th between 3rd and Lexington avenues (photo here. The building was demolished and in 1928 Proctor opened "Proctor's 58th St. Theatre." I'm not sure if there's a connection between the two. Quite a few establishments such as restaurants and hotels had "palm gardens", rooms filled with palm trees so the names may be just a coincidence (I've found no indication that Proctor was at all associated with the 52nd street venue).
 Contesting Castro: The United States and the Triumph of the Cuban Revolution by Thomas G. Paterson. Oxford University Press, 1994.
 John Coltrane: His Life and Music by Lewis Porter. University of Michigan, 1998.
 Malcolm X speaks: selected speeches and statements by Malcolm X and George Breitman. Grove Press, 1990.