Good question! I'm worried that the show is going to go the "alternate universe" route. I'm sure there are holes in this theory, but I feel as if the hints are adding up:
The fact that, as mentioned above, the narratives don't seem to work as either private memories or self-serving testimony.
Noah's speech to Alison in Episode 2
Alison's reference to "Sliding Doors" in Episode 7
This interview with Joshua Jackson, in which he ALSO references "Sliding Doors" (hmmm) and teases about there being similarities between The Affair and Fringe: http://blogs.wsj.com...out-the-affair/ He also says this:
At first I thought this was a reference to Noah's Episode 2 speech. Then I thought it was referring to the interaction between Scotty and Whitney. Now I'm not sure.
Again, I can't really back this theory up with confidence, but I have a feeling... Maybe I'll take this over to the speculation thread.