Living in a somewhat rural place, I find it hard to get 3 bids
sometimes as well, plus I don't like to keep calling and rejecting
contractors that might be wanted for other projects later on.

The best contractor I have dealt with did't make me ask for references
(which would all be rosy, as mentioned). On day one, he simply gave me
a sheet of literally over a hundred names and addresses all over my
area, without even being asked. Sure, he would have erased the problem
jobs, but still it was an impressively long list for a small
contractor, and all fairly recent jobs. He even went to the trouble of
pointing out which of the references would be most similar to mine
(down to house colors, trim details, style, etc.), and then also asked
right up front if he minds if he puts our name on the list as well, and
did so when the job was barely even started.