How much difference will it make to mix and match my USB chargers?

Some are rated 1A and others as low as 0.1A.

Can I just leave the 1A one in the socket and plug whichever USB lead and
device I want into it without worry?

--
Adam