What state are you in, you better learn the law of Adverse Possession
and find out how long he has been there and the building itself, and if
he put it up or the previous owner, because he may OWN the encroachment
and more if he has maintained it. He could have moved in yesterday and
be "Tacking" on the previous owners desire or not . Laws might be from
5-30 yrs, in my state its 10yrs for Adverse Possession to be complete.
Find out your states law and time limit and post back, you could be in
trouble if he has a claim and is willing to enforce it so removing the
building is not a bad idea now. Make friends now and try to feel him
out. Im involved in that crap right now, you dont want to go there if
you can avoid it. You need his survey , the county surveyors office or a
surveyor will have one.

You are going to get alot of wrong- bad advise here on this, and you
have ,such as removing it before you research the facts and know the
laws.