The real problem you are experiencing is a bad damper that is allowing
the cold in. there is an item called a Chimney Balloon that you blow up
in the flue to seal it off. This will insulate the damper or even serve
as a replacement for your damper and give you a tight seal that will
allow no air passage. It looks like they run for about $40-$50 online.
i use the clear variety of chimney balloon since it stores up in the
flue out of sight.