Had a Technics SA-300 with no FM at all. Ran that down to this IC which
is the final limiter and detector. Also handles the muting which I think
it stupid to put in that chip. But I prodded a bit to make sure some
dumbass 100K resistor isn't open somewhere.

Your chances of getting an AN377 are about as good as the lottery. So I
used an NTE788 from our wonderful ESI out in Mrntor, Ohio. Now I got FM,
which wss not before, but it is severely overmodulated. Nothing in the
service manual indicates the coil alignment would affect gain. It is as
if this chip was designed fro 25 kkHz deviation rather then 75,
seriously. And how it connects to the tank circuit at the end makes me
think it might be a slope detector, which ain't the best thing but maybe
they fixed that somehow.

If anyone is familiar with this chip, or Technics of this vintage, I
want your opinion on this NEW chip being bad.

I can't tell for sure, but it looks as if the NTE788 may be of the
similar design to (or even pin-compatible with) the Sanyo LA1230.

If so, it's a single-coil quadrature-detector.

I had a problem somewhat similar to yours with a family of
amateur-radio narrow-FM receivers a few years ago... the IF-detector
output started clipping at only about 2 kHz deviation, rather than
accepting up to 5 kHz or more cleanly. This sounded bad and
completely messed up the DTMF decoder.

The problem did turn out to be the coil... not its alignment, but its
in-circuit Q. The detector's output voltage depends on the amount
of phase shift introduced by the coil, at any particular amount of
frequency deviation, and the phase shift depends on the coil Q.

We talked to the manufacturer, and found that they'd had to change
suppliers for their detector coil a couple of years earlier when
the original coil was EOL'ed. Apparently the new coil they chose,
generated more phase shift per amount of deviation due to a higher Q
in the circuit. The Q is controlled by the impedances of the
reactances in the coil, and by the value(s) of the resistors(s) that
are shunted across one or more of the coil windings. My guess is
that the new coil had a lower L and a higher C than the
original... same resonant frequency, lower impedances, and thus the
existing shunt resistor had less effect.

We experimentally bridged the shunt resistor with a second one,
dropping the resistance and thus swamping the coil Q to a lower
value. The IF detector output dropped by about 50% at any given
amount of deviation, and the clipping was greatly reduced. A bit of
further experimentation and we were able to find a resistor value
which allowed the detector to handle 5.5 kHz of deviation before it
began to flat-top.

So, the problem turned out to be a one-resistor substitution in each
receiver... audio quality was greatly improved and DTMF detection
became reliable.

So, I'd suggest that you try an experiment of that sort... find the
resistor which shunts the coil, and decrease its effective value by
2:1 or 3:1 (just bridge it with another, for now) and see if this
un-clips the output.

The LA1230 data sheet comments that the chip's THD "depends on phase
linearity of the phase-shift circuit used"... I think this is alluding
to the same basic phenomenon.