You should have two wall switches; one on the red and one on the black.

Perhaps you are simply using the wrong switch. Either try to find the
switch on the black wire, or attach the red feed to the black fan wire.