Side view:

W| O
A|______/
L| /
L|/

Make a bracket that comes out from the wall, then goes up to the bottom
of the pole. Since clothes-hangers only contact the top and sides of
the pole, the bracket won't interfere with sliding the hangers. Just
make sure it fits in the space above the top of the hanger arm.

Dave