Chris writes:

Any suggestions for getting rid of these bees?

If you're up north in a freezing climate, they'll just die off around frost
season, at which time you seal up the hole(s). They live off flowers and
starve in the fall when the flowers disappear. That's why they're getting
so ornery around Labor Day picnic time and going after fruit-scented soft
drinks. The queen hibernates to survive over the winter and starts a new
colony in the spring.

So simply waiting is very effective.

The petroleum stream in a can is very effective, but it won't get inside
your structure to kill the nest. They can be nesting quite a ways away
from the entrance, so even tubing in insecticide is not always effective.

A Raid fumigator would do it nicely if you can cut a hole into the space,
which is a bit challenging if they decide to come after you for breaching
their enclosure. Those things will kill everything in a good sized room.