I guess I need to invest in a set of gauge blocks or make some sort of
jig to hold the bore gauge square with some of my mikes.

Just use your vise, wrap a shop rag around the shaft and clamp LIGHTLY so
you can set it with a mic. That will hold the bore gage steady to find the
smallest reading.