Caulk isn't very good for having water push it out of a hole/crack
like that (it'd work better on the outside of the wall).

How about patching with concrete then painting over with leak-blocking
paint? I got some of that paint at HD and it seems to work pretty
well.