When you give someone or something up, it doesn't mean to put it/them on the proverbial shelf to look at every now and then when things get boring.
It doesn't mean you should keep them in the background of your life so you can wander out to them when there's nothing going on in the foreground.
There's nothing uncivil about removing people or things from your life.
I'm not going to give any more of my attention to certain people and all the vices of my past.
Holding onto a piece of them builds the bridge to bring them into my present, and I don't have time to be tempted or distracted from the things that matter to me the most.
If that's cruel, so be it. Some bridges are meant to be burned.