I was thinking there might be some kind of venturi effect where air flow
will tend to pull in air from around it.


Why bother? When there's air flow in the ductwork, it's under positive
pressure, hence no air infiltration. When air flow stops, the pressure
iside
and out are the same, so still no infiltaration. HTH

Joe