The impulse energy is spread over the surface area of the rifle butt,

It is also spread over a longer period of time. The energy that your
shoulder absorbs is equal to that energy which is imparted on the bullet.
The bullet continues to increase in speed (pick up more energy) until it
leaves the barrel.