He is not a “good man outside of that”, he is straight up garbage BECAUSE of how he uses and manipulates you for sexual acts.
Dump him and move on. If he has issues with sexual morality then he needs to work it out with a priest and a therapist.
He is cohercing you into sexual acts. I don’t know the whole of your situation but it sounds like assault. Dont stay with him. He’s trash