snip
but I am attempting to
get a handle on the *physical* reasons why so many here say that a
drill press shouldn't be adapted to mill use.
snip
2 major problems:

(1) you have side load and interrupted cut on an end mill which a
drill press is not designed to take. At best the quill bearings
will get beaten out quickly. At worst, the end mill will slip in
the chuck, the chuck will come off the arbor, the arbor will com
out of the quill, generally at high speed.
(2) The lateral location of the drill press head is not rigid.
I.e. most drill presses are made so the drill press head and
table clamp to the column. Again, not a problem in drilling, and
in fact it may even be helpful to allow the drill to seek center.
Milling side loads will cause the head/table to shift resulting
in part/machine damage and possible operator injury.

If you are bound to try this, or have economic constraints, be
sure to keep everything as tight as possible, re tighten the
chuck frequently, use slow speeds and don't force anything. In
my not so humble opinion, it takes *MORE* skill and experience to
successfully mill with a drill press than a milling machine. Of
course, by the time you have acquired the necessary skill, you
wouldn't do it.

It still beats a file.

Uncle George