But is that what it does when it's heating? Does it chuck cold air out of
where it would put out warm air when it's cooling (IYSWIM)?

Exactly right, hence the term "Heat Pump".
The down side is that they generally don't work at low outside
temperatures.

John