Males speak of a place where only men may enter, and only the most powerful of men at that. Each guest must show his mastery of a drow female in order to gain admittance, and the members of this club are the secret masters of the city.
Just a rumor - such a club could never exist. And even if it did, it is just an idle club of wastrels and perverts, not a power in the city. Anything else would be blasphemy!