I had wondered if it had happened at Harrenhal and Ned accidentally walked in on them, when going to visit his brother's tent for some reason - I could have sworn I remember reading in a Barristan chapter that Ashara was dishonored at Harrenhal. I could have imagined that though, it's been a while since my last reread. I'll have to search my Nook.
After readers learned more about Brandon's nature, that Ashara had had a baby, and that Barristan loved Ashara and still considered Ned honorable, Brandon + Ashara made a whole lot of sense to me; she wouldn't be the first teenage girl to politely dance with the shy, quiet younger brother but decide that she really wanted the one who was brave enough to talk to her. (That would give a new dimension to Ned's "Never mention that name to me again" to Catelyn, when she felt brave enough to ask if Jon was Ashara's. Gosh, poor Ned. No, you can't have the pretty girl you danced with, but your brother can. Also, here, marry this redhead who has no idea your brother probably made the eight during their betrothal.)