Yes, in a very general sense we all have to conform to the act, but
if he builds far enough away from the boundary then the party wall
act has no relevance to his situation and he does not need his
neighbours permission to build an extension.

He never needs the neighbour's permission for the extension. The neighbour
can withhold permission all he likes, and you can still build right up to
the boundary line. You can even get an injuction allowing you to enter their
land to do so under some circumstances.

Christian.