Its funny, the main breaker is no more dangerous than an outlet, but it is
somehow terrifying. I still get nervous when I check for tightness every
now and then.
It is a very simple to replace the main breaker, but if you are not
comfortable with it, then get an electrician. It shouldn't take more than
30 minutes.

It might just be that the breaker is not seated properly, or that one of the
contacts is loose. You "might" want to check those first.
If you 1) wear shoes, 2) keep one hand behind your back, 3) wear rubber
gloves, 4) are careful not to touch anything you don't mean to, and 5) use
an insulated tool, you can't go too far wrong. (1, 2, and 3 are completely
unnecessary, but they protect you in case you screw up on 4 or 5)