Out of curiosity, is there anything wrong with leaving the ends (rake
board and rafter ends) un-cut and un-plumb and just nailing the fascia
board on that way? Granted it would look a little unusual but for a
garage or shed or fishing camp etc, who cares?.... but is there some
structural reason why it needs to be plumb?

There's no structural reason I know of. Lots of buildings have been built
this way.

As Duane said, a plumb fascia makes it easier to attach gutters. Though
there are gutter brackets that fasten to the roof to "hang" the gutter
from above. Or, you may just eliminate the gutters altogether if you put
rock around the building or otherwise reduce splashback onto the
building.

Just from a practical standpoint, you aleady have to plumb cut the rafter
at the peak, so it's not much more work to plumb cut the tail. If you
have a power miter saw, it's the same angle. Cut, slide it down, and cut
again. Easy. Cutting the bird-mouths where the rafter sits on the wall
are more time consuming.

Anthony