Please speak up. You don't need to be "something" to speak up against prejudice. You don't need to remind them that you're Jewish, they shouldn't be making these remarks. And having had this experience, I hope it helps you speak up when other minorities are disparaged by your friends/family. Easier said than done, I know, but ignorance is even easier.