People like detail on large objects. Without it, we have no sense of scale - the physical range-finding made possible by binocular vision is about as good as a cheap camera's focus, with everything beyond ten or twelve feet registering as "far." We use details we know to internally calculate distance, and this is learned very early in life. We come to understand that those little people we sometimes see are normal-sized but located far away. (Those who see actual little people, whether leprechauns or elves, are outside the scope of this blog.)
Traditional architectural styles provide a lot of detail for the eye. A common paradigm is that a building has three scales of detail: the overall shape (or "form" as architects insist on calling it), large detail, and small detail. The idea is that the building's shape can be read at the limits of vision, the large detail can be read from a distance, and the small detail provides scale when at close range.
A traditional example, 17th-century St. Paul's in London:
A modern example, the 1929 Chrysler Building in NYC:
Closer, you see the black-and-white brick patterning and more details of the windows:
At very close range you see details of the brick and gargoyles:
and even seams in the stainless steel skin:
So if both traditional and modernist architecture give us multiple scales, what's the issue? Several types of modernism, most notably International-Style Modernism, tried to get rid of detail.
The 1958 Seagram Building in NYC
More recent examples are even more scaleless (less scaleful?). The two 1970s UNDC buildings in NYC, across the street from the UN, have nothing to relate their size to a viewer:
The problem with "glass boxes" isn't that they're sheathed in glass or that they have boxy shapes. The problem is they do not relate to human scale because they have too little detail. I've never heard anyone say they felt the Chrysler Building is too big, even though it's noticeably larger than the UNDC buildings.
Blame Mies if you want, blame Adolf "Ornament is Crime" Loos, but most of all blame developers who seized on an architectural philosophy of simplicity and use it to justify cheapness.