This is what I thought, too. Add to that, Sherlock was focusing on so many things (Moriarty, other cases, etc.) that maybe John thought he was distracted from his focus on Mary and therefore hindered Sherlock's ability to keep her safe.
On the "affair" - maybe I missed something (I was watching with someone who wasn't following at all so had to translate here and there, so I was a little distracted) but it seemed to me to be ill-advised flirting that was setting up something to come. I'm not sure that I would call it an emotional affair - he and Mary were exhausted, Sherlock was hyper, lo