And you might have to take a look at yourself as well, are you giving him the cold shoulder to often, giving him attitude when it isn't needed and so on? Not saying that it's you, but it's something you have to look at.
If you can honestly say that you've done everything in your power to make it work and nothing is changing, I'm talking EVERYTHING even couples consoling. then you can leave him. Like the others said, don't deny him the kids, if he doesn't make the effort they will understand when they are older. Kids need two happy parents. Not two parents to just have because "that's what they need."