As for a pole barn, we don't need permits for those here as long as
they are for agricultural use only. Last time I was at the inspection
office, they even had free plans for pole barns. But if you park your
cars or RV's in the "barn", it's no longer considered agricultural
and you probably need a permit

Who would know what you parked where?

I'm just relaying what the building department told me...

Obviously, you could build the pole barn for "agricultural" purposes, and
park your vehicles in the "barn" after it's built. Just be aware the permit
requirement may be based on the "intended" use.

We built a small house with two bedrooms and a home office. It even says so
on our plans. But to the building department (and taxes) it's a three
bedroom house. Even though there's no closet of any kind in the third
"bedroom". Though obviously, we could move out the desk and put a bed in
there at a future date.

So, your building department may view things differently than you do...

Anthony