When I was faced with exactly the same problem as you on soft red
rubbers which had been painted over by my previous owner so the outer
part of the brick had absorbed some of the paint and there were
pockets therein painted -I considered lots of options and tried
samples of all sorts of cleaners chemically and manually. None worked
and in the end I had no choice but to get to it with a dirty great
angle grinder with a face disc on it and took off the guard so I could
VERY carefully take off the outer 1mm or less of the brick. I was
told this would ruin the "face" of the brick and it would all degrade
in a few years. I could see no logic in this statement as the
material of the brick was not different on the face and 05 or 1.0 mm
in.

I've never understood that either and my experience is the same as yours.