On Sunday, March 29, 2020 at 11:03:03 PM UTC-7, John Doe wrote:
I tried a hole saw in my cheap drill press. Works perfectly for
hardwood, but doesn't work for 6061 aluminum

It's the per-tooth load that's your problem; the drill press probably doesn't
have a low enough speed for proper high torque on the hole saw, so the
per-tooth cut can't get deep enough. I'd suspect you're getting fine
waste crumbs instead of proper chips. Lube is important for aluminum
in this kind of thing, too.

The 'remove a few teeth' might help, or resharpening the saw (to make a slightly
less agressive cut) might do some good.