The Upper Classes are all about the semblance of propriety and "niceness." This IS the North and yes, Black people who probably were never enslaved will walk amongst them. The Upper Class Whites are fine with that because there are SO FEW Black people and very few Black people of means (dressed correctly and able to mix socially). Honestly, we are only 13.6% of the total US population TODAY! I have met many parents of my schoolmates while growing up. While my parents fit in with the Upper Classes, I was frequently the only one or one of two at social gatherings and of course at (private, nonsectarian) school. It was the same in (private) college. The parents and children seem to have the freedom to accept you as they would anyone else. Why? Because Black people (and especially ones with money) are not a threat - there just aren't that many people like me in those spaces.
Later in life, the only racism I encountered personally came from people with very "new" money or in lower classes. It sounds weird to talk about in 2022, but it's true. I DID threaten these types of people for some reason and they had not been raised with the same social customs as the Upper Classes (and old moneyed). So, it is not odd that Peggy isn't being insulted (or grossly threatened) to her face. Now, if she were in the Midwest or South, she might need to be chaperoned because rapes were a distinct problem for Black women of some means after slavery until the 1930s (documented in newspapers and books). I think this might be why the Scotts want Peggy safely married and around her own environment. Even today, Black parents of all stripes ALWAYS keep a watchful eye out for their children. That is one of the major reasons we have survived.