According to The Ranger :
I was at the side yard, opening the gate to put away my mower when
two of the largest bees I've ever encountered flew out from inside
a knothole in the lower portion of the 4"X4" post. They weren't
too aggressive, simply floated at my eye level and watched me
until they'd decided I wasn't a threat. One was blonde, the other
black. Judging by what I've found through google, the blonde is a
male and the black female.
http://www.whatsthatbug.com/carpenter.html

Since I'd rather _not_ get stung removing the fence post (it
cracked loudly when I opened the gate), is there some preferred
method for evicting said colony?

http://www.uky.edu/Ag/Entomology/ent...ruct/ef611.htm

The males do not have a sting. Females generally won't
sting except during nest construction or if you outright handle
them.

I do not think carpenter bees make colonies. They're solitary
bees like bumble bees. If you really have to, a single shot
into the hole with creeping insect killer would probably
do the trick.

Given how tame these bees usually are, with a little care
you might be able to remove the post without killing them,
and park the post somewhere out of the way. Try this
on a cool night when they're lethargic.
--
Chris Lewis,

Age and Treachery will Triumph over Youth and Skill
It's not just anyone who gets a Starship Cruiser class named after them.