Yeah, he also called it “a game of flats” which is apparently a term for lesbianism during the 18th century.
He very clearly wanted Charlotte because he thinks his sister wants her also. Lady Fitz calls him on it to get under his skin when she says, “you seem to think I have some unnatural lust for Charlotte, perhaps I should find out”.