Just thought of a possible easy, cheap and fast solution to your
dilema. Pull off a piece of something with the exterior paint color.
Go to a paint store and have them match it. Or use a can of the
existing paint if you have some and it hasn't been so long since it was
painted that the color has faded a lot. Then touch up the caulked
areas.

I think you are then in a much better position. I think now it's
reasonable for you to claim you correctly made the repairs that were
called for in the inspection. At that point, I would just follow the
contract and not do or say anything more.