If they'd leave me alone it'd be fine, but
any time I want to drill the wall or just get up a ladder in their
vicinity, they get rather aggressive. That's when I get all on their
arse back.

A few years ago I was up a ladder re-pointing round a wasp entry hole.
The first hour was scary, but then they saw the benefits of using my
chisel as a landing stage. By the end of the day we were just ignoring
each other and going about our business.