You will never get it all off better off leaving it. Brick is usualy
painted for a reason, it is going bad.