Easy - NO!

Straightforward yes, chip out loose material if that includes the plaster undercoat take back to brick. PVA the brick. Fill with bonding plaster or if you can plaster OK then use one coat plaster smoothing to a finish alternately apply finish plaster to bonding coat. If you are not confident with finish plaster use my favourite filler namely drywall filler. You should be able to smooth it out well but if you do not it will sand down a dream.

Richard