snip
Hmm ... consider the possibility of using a Morse Taper end mill
holder which matches the spindle nose, and making a drawbar to hold it
firmly in place. (For that matter, A Morse Taper collet could be used
as well.
snip
A morse taper collet is certainly usable but because you most
likely will be changing mill cutters several times in any
project, it will be much easier to use a end mill holder as this
will require only the use of a allen [hex] key from the front of
the spindle.

A morse taper collet must be have the draw bar slackened, tapped
forward in the spindle, the end mill changed, and retightened.
This gets old very quickly and there can be a tendency of the
endmill to shift in/out of the collet.

For typical home shop machining you will only need one or
possibly two holders 3/8 and 1/2 inch.

Uncle George