If you get better results when the drill press is running I think you
have excessive runout in the quill.
Put a long bit ,largest diameter the chuck will take, turn drill press
on. Do you see any wobble?
If you have a dial indicator and a length of drill rod, you can check
runout better.Sometimes the chuck can be at fault, though probably not.
mike