The offset you care about is the center of the bit relative to the
center of the bushing. The offset from the finger to the bit is
irrelevent here - the jig is designed to accomodate *that* offset.

When pondering your diagrams, think to yourself "the bit is to the
left of where it should be".