I'm still wondering why you allowed him to dump the remainder of the
stone even after you knew it was wrong the size.

People get flustered. He didn't know what to do with the jerk driver
insisting on dumping the load, so he just let it happen hoping for the best
somehow.
He probably isn't the confrontational type; give him a break.