Any chance the pad loaded up with galled aluminum, so that you were
basically trying to polish aluminum *with* aluminum?

Abrade or machine it flat and start over...