If Fred is considered the equivalent of a secretary of state or minister then only having him meet deputy ministers and someone from the PMO could show that while Canada is willing to listen, they're definitely not giving them the curtesy due recognized officials/nations. No state visit, no PM, no governor-general, no meeting in Ottawa. They probably wanted it as under the radar as possible.
I also agree with everyone who said it makes no sense that the Canadian government would have them there in the first place considering what is public knowledge about Gilead. Both Moira and Erin were Handmaid's, plus Moira's sex slavery, so it's not like the information in the letters would have come as a surprise.
I was rather hoping Moira was going to shout something at Waterford to show that she knew who he was, and so Serena would find out about his extra-curriculars at Jezebel's.
It's too bad the show didn't film in the summer so that the people on the street could have been wearing brightly coloured clothes instead of mute winter greys and blacks. That would have been such a contrast to Gilead.