This is totally what I think! And I believe Pete doesn't even realize he's holding a grudge against the little girl who accidentally shot him.
We need more backstory on Hetty, Trevor, and Sasappis. I want to know how each of them died at the manor or on the grounds. Aside from Flower, Pete, the cholera pit ghosts, and the headless guy, no one else has visible injuries.