I was surprised to find out that Peter was 34, assuming he's not telling fibs that is. Maybe it's the grey hair but I was sure that he was already in his 40s so it did strike me as odd and slightly amusing that he was labeling someone who is still in their 40s (49) as old.
Middle age is usually considered to be around the mid 40s to early 60s. If Peter's claim that he was born in 1980 is to be believed, he is just 4 or 5 years older than Liz, who is almost 30, who was the other castaway involved in dismissing Debbie and Joe as the old timers who are bound to hold the tribe back in challenges. He isn't a spring chicken but middle age would more aptly describe someone like Debbie.