If you want to get rid of the mouse for good your are going to have to
kill it. No point pussy footing around (excuse the pun). You rarely
get a solitary mouse, there'll be more if they haven't arrived already.

Get plenty of lethal bait and exterminate the lot, follow the advice
others have given in covering as may holes as possible etc.