Dean gives a ton of references to Mary in the early episodes, so I really disagree that Castiel is so high above his mom. Mom was a subject of repeat conversation throughout those early episodes. In Advanced Thanatology, look at Dean's reaction to the boy saying "I miss my mom". Notice he asks about his mom, not Castiel, to Billie. In the counseling scene, it's all about Mary. When Castiel came back, that sparked hope for Dean, which is why he had the mood change. He is really great friends with Castiel, not denying that, but if he was so heartbroken over Castiel himself, then he wouldn't have been fine with not talking to Castiel and not caring that they don't see him for such a long period. He wouldn't have been so easily fooled by a fake.
The brothers switch moods based on their perception of Jack. Dean was scared of Jack in the beginning. When Jack brought Castiel back, Dean no longer saw him as this huge scary thing and had hope again. In Tombstone, Sam starts getting scared of Jack. Notice his little head tilt when Jack demonstrates his power and again in the closet when he's talking to Dean.
Sam has been showing fear of Lucifer for awhile. It was in background in 11. He tried to put that aside for Jack (although you can tell he's scared in 13.01). In Breakdown, there's a ton of hell imagery. Sam thinks that his mom is stuck and being tortured by Lucifer, so of course he's really depressed.
He knows something went wrong with Jack's rescue mission.