I've got the same concern now that I had AC installed last summer. My
plan, which I have not implemented yet, is to build a thin insulating
panel - probably some paneling (white to match the ceiling) with some
weather stripping that I can attach to the ceiling in the hall to cover
the fan and stop the air leaks.

Also had a recommendation years ago to build an insulated "box" to go
over the fan in the attic. Rig a pully to be able to lift the box off
of the fan from the attic access since it is a pain to crawl over to
it. I'll probably just throw some rigid foam insulation over it and
forget about it. I have a feeling I won't be using it much next
summer, but don't want to bother having it removed right now.