I'm pretty much agreeing with you here. The only difference is I'd add Princess Alice to the mix; whom I found fascinating. But of course, since she's of advanced age at this point in the story so we had precious limited time with her.
Other than that, I just found it really hard to care about Margaret, Charles, Camilla and even the political storyline.
Season 3 was fairly disappointing for me.