Some have said that on *really* heavy work and end mill can be
"sucked" out of a collet - a small amount, but enought to ruin a
precise job. I have not seen this, but also don't anything super
heavy.

One advantage of holders is that you would not have to mess with the
drawbar. On my mill I have to flip up the pully cover to change
collets. Having said this, I still use collets - I'm not that lazy yet
;-)