Ben-

Did you use premixed compound (comes wet in a tub or box) or did you
use fast set (mix w/ water)?

Sounds to me like the compund is just not dry (could take a long time
ifthe material was really thickly place) or it is indeed lightweight
compound (lower density)

At think point you could just not paint & see if it hardens up.

I would roughen up the painted surface & cover the whole area with a
water mix compound, behave more like plaster.

I would suggest against futher painting until you get the finger nail
preformance you desire.

Whe you let the coats dry did you get shrinkage cracks? if you did
then it was most likely dry, if no cracks it either wasn't completely
dry or ?

cheers
Bob