Whether Varner is correct or not (& I hope he's not), it was the production company & CBS who chose to remove Dan, & cast members were not consulted so their opinions carry no weight except in the 'court' of public opinion. The company would not have done so without being damn sure that they had sufficient documentation to protect themselves against any action that Dan, or anyone else, might take for their doing so.
Of course, one wonders why it took so long! After all, they had, & have, tape showing that Dan to continue to touch women inappropriately for more than a MONTH, a company that 'warned' him a couple of weeks into the show, but did nothing, until a production member received the same treatment that contestants did. Something finally compelled them to act, & it must have been a biggie.