Eric,

Pumice is pretty soft and is marketed for this use. You can scratch the
porcelain noticeably if you use a lot of force or do this scrubbing
routinely. But you'll probably be ok.

Dave M.