I think I figured it out, at least for me. If the season is a season I enjoy with a good cast I tend to remember who was on it. For instance David vs. Goliath is by far and away my favorite most recent season. It had a great cast with Nick, Gabby, Mike White, Angelina, Christian, Davie, John, Carl, Lyrsa, Natalie. That is just a tremendous cast right there and you know you have a great season when people remember the early boots like Natalie (Who I hope they bring back) and Lyrsa. Same goes for Cagyan that was an amazing cast and I can name most of them just by seeing them on screen.
Then you have Edge of Extinction and besides the return player and a handful of new players (Reem ftw!), I simply do not remember that cast at all because the season and the concept of the season just was not interesting to me.