Question: it looks like you're using a small heavy duty bearing. Why
not one of the larger light duty bearings? They have a circular
outside shape, usually a thinner profile, and, in my experience, turn
easier and smoother than the heavy duty ones.