Not only should you certainly shut the air conditioner off, but you
should also go to your main breaker panel (basement or closet or
wherever) and shut off the entire circuit that the outlet is on, too.
If you are not sure which breaker to switch, plug a radio turned on
loud into the outlet (or a lamp or something) and then start flipping
breakers till you find the one that stuts off the outlet.

Really. You are ripe for an electrical fire. This is a major cause of
house fires.

Go do it now. I'll wait.

Ok, got the breaker off? Now check to make sure that BOTH plugs are
shut off. It is unlikely, but you may have a "split circuit", where
the two plugs are on different circuits (in which case they are
supposed to be on one double-breaker, but diy'ers screw this up all the
time). You want NO power to that outlet box.

Now finally, dont turn that breaker on until you have the box looked at
by an electrician. A DIY'er can diagnose these things, but you say you
"know nothing about electricity", and today is not a good day to start
fiddling with wires. So call an electrician. They might recomend a new
run of wire, or upgraded circuit, or all sorts of things. You might
also ask them to poke around some of your other outlets and wiring.
Overhead lights are a cause of problems, in particular (the heat ruins
the wiring).

-Kevin

BTW -- if you haven't shut the breaker off, I wasn't kidding. I'm won't
stop writting until you do it... Well, ok, it's your house. Nevermind.