Mrs. __B and I spent yesterday on a tour of house museums in Queens. One building, the Bowne House, was built circa 1660 and is either the oldest or second-oldest building in NYC. It is, of course, a farmhouse.
One of the strange consequences of living in an area colonized after the "age of exploration" (my god I loved my 5th-grade history class) is that the oldest remnants are farmhouses and a handful of house-sized churches. Nothing much bigger was built back then except a few mostly-earth forts; the longest settled areas are mostly city centers and have been rebuilt multiple times. The Bowne house survived because (a) Flushing wasn't heavily built up before 1900 and (b) the same family lived in in through nine generations from 1660 to 1945.
In longer-settled areas (say, anywhere in Europe) the oldest above-ground remnants are big: castles, forts, pagan temples, cathedrals. Anything smaller had been eroded by thousands of years of social friction.