Ah, this could explain it, though I was following the directions on the
rawlplug (which were for brick) - and using the specified size of
masonry bit.

I've also been told to put some welly behind it to get it through the
plaster as fast as possible and into the brick, so it doesn't dislodge
more plaster than needed. Not tried that one yet...