For me, this entire season has been about Loki's growth, with some setup to Phase 4. Loki plucked from the losses in New York would have easily said yes to Miss Minutes' offer in the beginning of the finale. But instead, the TVA, and more specifically, his uneasy friendships with Mobius & Sylvie, showed him a different way.
Also, I think I'm one of the few who didn't really see a "romantic pairing" between Loki & Sylvie. Sylvie, unless there's some comic book reason she's not, is for all intents and purposes Loki. Another version of himself. So yeah, I get that he would ulti