In seasons 3-4 everyone played in the veto. They changed to 6 people in season 5. I think it's for practical reasons. You can't have 16 people playing OTEV, spelling comp, or most of the games they use in recent seasons, in the backyard space. It would have to be one-at-a time type of comps and I don't think they will ever go back to that.
In seasons 5-6 HOH and noms picked the players. The current selection process has been in place since all stars for the reason you mentioned. That is not going to change either.