I am just wondering how others would deal with this matter.

Send them a registered letter stating that you have still not recieved the
3" that you ordered and that the driver insisted on dumping the gravel
against your wishes. Then tell them exactly what you expect them to do
"waive the bill","Come get your gravel", "Deliver the 3 inch", "cut the bill
in half", etc..

At that point you have legally made them aware of the issue and it's up to
them to make the next move.