I think in that moment, Rick may have thought that he deserved it. He had just killed his partner and best friend. Take away all the mitigating circumstances and stand there with him. He had just put a knife in the gut of one of the people in the world who knew him best. And was okay with it.
Then, he turns and sees his impressionable young son standing there as a witness. All the reasons for killing Shane - which were pretty darn good and long overdue, in my opinion - fly out the window because his kid had just seen him murder a person they had all known and loved.
When Carl raised the pistol and Rick began sputtering, I can see just a moment where he thinks to himself, well, damned if I don't deserve this end. When Carl shoots past him to put Shane down, he is confused and it takes a while for him to realize what he had just seen.
On another note, I blame Lori for Shane flipping out. After the battle with Rick while in the police yard, Shane seemed to have come to a resolution regarding his relationship with the Grimes family. If given enough time, he would have left the group to start over elsewhere, possibly taking Andrea with him. When Lori went over to him and "made nice", it slipped his gears and he went nuts. What on earth was she thinking?