In my opinion, driving it back in is a last resort. If you can remove
it once the screw pulls the wall back, by all means do so. If it was
inclined to stay put, it would have.