With oil, I don't bother. It just doesn't seem to make any difference.

With varnish, I use a cloth dampened with mineral spirits.