Most likely, you're dealing with part deflection. It bends away from the
cutter, making it larger the further from the chuck,

Very finely sharpenned and honed tooling helps a lot with this issue. if
you've got it, super high RPM and a light finishing cut help too.

Karl