He was responsible for the other things you mentioned but In Episode 8x15, it was revealed Simon was, in fact, responsible for the slaughter of the Oceanside community's men and children, rather than Negan. "When you helped me take this place, I wasn't sure I wanted to keep you on board," Negan said. "Before me, before there was a system in place, what you did, killing all those men, those boys in that settlement so long ago... A lot of people would think that is some psychotic s---! Like that is the work of a broken, demented, g--d--- ghoul!"