I always use DIF/hot water and those green scratch pads, and haven't
had too much trouble. Seems maybe I haven't run into what you've run
into.

As for painting, you could always prime with an oil-base primer. I'd
still try to get the heavy spots of glue.