Dave:
Since you are working with such a thin wall on your tubing, I would
suggest trying a 1/4 inch end mill. You can chuck it up in your drill
press since it is hardened, it may slip if it hangs, but if you get a
center cutting one, it may resolve your problem. I would suggest a 4
flute end mill. If you can resolve the slop in the drill spindle, that
would help. I drill a lot of 2 inch square tubing for trailer hitches.
We use an annular cutter in a drill/mill. also, be sure you are not
drilling thru the side where the seam is. It is hardened and thicker
and will caouse drifting of your drill.