There is a fireplace insulator called a chimney balloon that is
installed usually above the damper and inflated into place. This would
stop the heat release issue through the chimney flue but I understand
you have an issue with heat and cold actually conveting through the
brick itself.