I guess we have reached an impasse, but I can't resist having one more go at it. 🙂
Dean's lack of hope that they would find another way cannot be equated with a lack of hope that he could still get people into the lifeboats. Dean had hope that what he was planning to do could save people, just as Sam did. Except if Dean was wrong, a lot of people might die, while if Sam was wrong, everyone might die.
The difference in Dean saying yes and Sam saying yes was not that Sam's plan was better because he had more hope. It's that it was a plan based on the fact that they now had a way to trap Lucifer, which they didn't before. The writers could just as easily have had Dean then say, "Great! Now we have a way to trap Lucifer, and no one else has to die! I am going to say yes to Michael, take control over him, and make sure Lucifer falls into that hole!" The only reason -- the ONLY reason -- that this didn't happen is that the writers had decided to take Dean out of the story at that point, and Sam was going to be the hero to save the world. It had nothing to do with Sam being more hopeful or his plan less risky.
Remember, in comparing Dean's considering saying yes with Sam considering agreeing to Ruby's plan, an important point is the word you yourself use in describing Dean's choice: "potential". This wasn't a mathematical equation: remove half the population, and the other half will remain. And I think this is where Dean was coming from a different place than Sam was in considering Ruby's plan. Potentially, if Dean said yes to Michael, half the population might die. Or maybe more. Or, if things turned out differently, maybe less -- or maybe no one else would die. If Sam said yes to Ruby, Nancy definitely -- not potentially -- would die. She had to die, and at Ruby and Sam's hands.
So why was that such a big deal, if we are only talking about the loss of one person? Because it wasn't just about one person. Because of what ILoveReading said:
Ruby was trying to condition Sam to obey her, to trust her judgment, to do what she said when she said it was necessary, to believe that the ends would justify the means. To believe her when she said that it was all up to him to save everyone, even if it meant going down a dark path. To feel that he had gone down that path too far to make it possible to come back.
And, of course, Ruby succeeded. She didn't succeed in making him kill Nancy, because Dean stopped her. But it was a step in her conditioning of Sam, that he was willing to consider crossing over that line. That's why Ruby went to them afterwards to rub in the fact that people had died anyway, to reinforce that conditioning. It was a first step toward Sam's decision to kill Lilith, resulting in the return of Lucifer, the beginning of the Apocalypse, and the death of a lot more people than just Nancy.
In spite of what you assert, that the writers were refusing to "condone" the risk that Dean was taking by deciding to say yes, I never saw anything in the text of the show, or anything in what the writers said, to indicate that they intended or wanted us to see Dean as about to step over the line that Sam was about to cross with Nancy. They absolutely would and could not do this. Why not? Because they knew that they were going to have Sam say yes -- at the potential risk of the loss of not just half the population but of everyone -- and they were going to frame it as a noble and heroic thing to do.