First of all, I'd like to repeat: make sure it is actually the tool rest,
and not the banjo that is off. An easy test would be to reverse the tool
rest in its holder, and see whether the slant reverses.

Second, I'm going to stick to my guns here, even though I seem to be in a
minority of one. The slant is only 1/4" in 12". Only if lcrumley is
turning long straight spindles will this even become an issue. I know it is
one of the "Commandments" of turning that the tool rest has to be positioned
close to the work. But it isn't even possible to follow this rule, again,
except on straight spindles. If you place the toolrest as close as possible
to the work, you minimize the force necessary to hold the tool in position.
That's plain leverage. But, you also minimize the amount of vertical
correction available to you. If you visualize a skew resting on edge, or at
an angle, while you turn a cove or bead, it is necessary to raise or lower
the handle to make the tool follow the contour of the work. Likewise when
you are turning a bowl. It is rare that the bevel will follow the arc you
are turning as you swing the tool, and roll the tool to produce the cut you
want. Generally, you also have to raise or lower the handle to keep the cut
you want. The closer the tool rest is to the work, the narrower the range
you can cover before you have to stop and raise or lower the rest.

From my own experience I believe the feedback goes from your cut through
your senses to your brain, which tells you to raise or lower your tool tip,
or rotate the tool, or change its angle. This is not an extraneous
experience that is introduced by the tool rest--it is an essential part of
turning that we all learn and practice.

Finally, in anwering lcrumley, I tried to put myself in his position. A
newly acquired lathe with some problems. Should he concentrate on
eliminating the problems, or can he get started turning, and have some fun?
I see no harm in going ahead and doing some turning while considering what
to fix first, and how to do it.