After killing off Jack, I know I won't be watching the show.
I know that it was DL's decision to leave and I read somewhere that Hallmark and the show runners knew about it a year and a half prior his last episode, which makes it probably around the time they were filming season 4, and may be that's why he missed the second half of the season too. But even in season 3, especially in the first few episodes of it, he had limited screen time (compared to seasons 1 and 2) with all the knew characters popping up out of no where. May be he wanted more screen time, may be he didn't like that his name in the credits went from the second spot to the next to the last one after 'with', which I know shouldn't be a problem since they say that that spot is as important as the second spot, but still. Or may be his personal reason's that aren't even work related, and he just doesn't want to talk about it.
As for Lori Loughlin's statement that the crew could lose their jobs if the show gets canceled because of his departure, one can also argue that they kept their jobs for five years because of him or because of the Jack/Elizabeth pairing.