Can anyone tell me whether I need to sand off all the yellowed stuff?

Doesn't sugar soap shift it?

No, I'm afraid it doesn't.

There _was_ a yellow film on the surface, but I have already washed
that off.

This is the paint itself yellowed.