"The lack of through traffic or modern industry, together with the borough's role as a fashionable summer residence, have preserved its colonial, Federal, and outstanding Greek revival domestic architecture, while the activity of one of Connecticut's last remaining fishing and lobstering fleets keep it from being simply a quaint, historic village."