I've also had my share of problems; they seem worse when I pick a size
that's too small. As mentioned, center punches and little drill bits
help in getting an accurate center. This helps being able to go
bigger.