I find that a kitchen scrub sponges (the ones that are blue scrub on
one side/blue sponge on the other) do alright.
you scrub up off as much as possible with one sponge till it is caked
with caulk, then toss it and try a new sponge