I've had similar issues in my bedroom and the front closet - both areas
are outside walls and my 1950s Michigan house does not have very good
wall insulation. In the bedroom I just make sure I don't let any
clothing, blankets, etc. stay up against the wall. For the closet I
try to keep the door open an inch in the winter to let some warm air
in. I've been meaning to put an opening in the door or wall to let air
circulate with the rest of the house, but have not gotten to it yet.

Wash the clothes, clean the wall with regular household cleaner - maybe
something with bleach to kill the mold.