Why? A competent estimator performing a manual J calculation will give the
proper size to use.

Because I've spent too many years living in houses with a/c units that were
sized according to the house size, and most of them ineadequate. When it hit
110+ outside, we sweltered inside because the a/c just couldn't handle the
load. I've had several of my units inspected by repairmen, and all of them
told me that the unit was working fine but just wasn't big enough for the
house. I had a 1200 sq ft house in Vegas, and a 3 ton a/c. It didn't cool
worth squat when the temp exceeded about 105. A contractor told me to
replace it with a 3 ton unit. I told him that the current 3 ton unit was
fine mechanically, but would not cool when it got too hot, and then he
suggested that because it's hotter in Vegas I might need a bigger unit.
Well, duh, why don't they do that in the first place? There is a big
difference between the size of the a/c unit need to cool a house in 90
degree humid weather, and 118 degree arid weather.

I wonder if the problem is that the contractors use the smallest unit they
can get away with, instead of one that will cool the house even when the
weather hits 118? IOW, a competent estimator would indeed give the right
size to use, but those that build houses like to cut costs when they can,
including using small a/c units?

Unfortunately a BTU/square foot chart is only an extremely course guide
and isn't really accurate enough to properly select equipment.

Agreed. A couple of computers, other electrical equipment, a souther
exposure, etc. can make big differences in the size of the a/c you need.