You're not taking a big bite out of the wood. I can't imagine that your hand control is not adequate for stable/controlled cutting. If the coplanar hypothesis is not the problem, then the only other things I can think of is a dull bit or it's turning in the wrong direction.

Sonny