Sounds like a job for "Wasp Nest Destroyer" to me.
At least that's the stuff I used after the council did half a cock up job
of removing a nest in my loft.
It's a foam sprayed from an aerosol and you should apply this to the
nest/nest opening each night when activity has stopped.
The type I got had a range of about 2m with reasonable accuracy - I was
working off a ladder so didn't want to get too close.
Yes you can leave them, but you may regret it if the nest gets as big as
ours (effectively preventing safe entry to the loft space where I was
trying to install vent fan and ducting).

Phil

These aerosol sprays are very good if they are causing problems or risks to
kids / pets, especially if the nest is inaccessible. Although they can be
annoying if you do a lot of eating/drinking al fresco they do predate a lot
of other insect pests.