"Maybe a few $500K homes mixed in with lots of $100K homes makes the
community more appealing from the POV of the people who own the $100K
homes."


Maybe? Of course it does. It would be nice for the person with the
$100K house to have it sitting together with $500K homes. Howver, as
has been pointed out, it's almost always a detraction from the value of
the $500K homes. If you have a neighborhood of $500K homes and
someone then built a $100K house in the middle of it, there is no
question it would detract from the property values and look out of
place. And it would be unfair to those with the $500K homes, as it
would decrease the value of their property and the asthetic appeal of
the area.

Like most people, I see nothing wrong with using zoning or HOA's to
maintain some consistency within a defined neighborhood. That's one
of the main purposes of zoning, to regulate what you can build and
where. It keeps commercial, industrial, and residential areas
seperate. And within residential areas, it keeps apartments and small
homes seperate from larger single family homes. You can still have a
wide variety of styles of homes, but it maintains the property value
and keeps the nieghborhood from becoming a hodgepodge of crap.