To me it looks like the vise might have been slightly unstable on the
swivel base when you milled it. I have cleaned up import milling
vises, my anvil, and a PYH end mill sharpener like this:

If the top surface wasn't milled, level it with wedges and take a
light pass to remove high spots and see how it cuts. Shiny spots are
harder and may be high afterwards.
Flip it over and block it up on the just-milled surface. Cut enough
off the base to see fresh metal all around.
Turn it back upright, locate clamps over the cuts on the bottom so the
casting doesn't warp, fly-cut the top. I take no more than 0.005" per
pass and often 0.001" for the last one, after stoning the bit.

I tried this on an RF-30 mill-drill once without success, the vertical
feed was too sloppy. I had to take the vise home to mill it.

Since I have a surface grinder I next grind the bottom side until the
wheel cuts at least most of the area, turn it upright and grind the
top to taste.

Previously I filed most of the tool marks off and then finished with
SiC paper wrapped around a file or a piece of scrap tooling plate. You
can file in the middle of a surface without rounding the edges with a
file that curved during hardening, which is common for the cheap ones.

I would clean up the swivel base of your beautiful red vise separately
on a faceplate in a lathe.

Jim Wilkins