I have cut aluminum with circular saws, both hand held and table. The
key is to use carbide tipped blades. I'm pretty sure that your plywood
blade was not carbide.

It will not cut aluminum as fast as it cuts wood (duh?) - so don't force
it. You can also make multiple passes, starting with the blade low and
raising it each pass.

It is much harder on the blade's sharpness than wood is (another duh),
so I keep a separate blade for aluminum.