Have you closed the damper?
Might also consider a chimney cap to help block downdrafts.