Do the shoelace test to find out where the smoke goes.
That will at least tell you what's happening.
Why it's happening is a matter of speculation.

Vent stacks act like chimneys in cold weather (or at night) because the warmer air inside them wants to rise, and that updraft could be drawing air out of your house through the open floor flange.