For what its worth (not much :nervous I don't think there are any performance hits but I can't say for sure. I am running 64-bit Linux (at work) with some servers having only 512Mb of memory. Most of the servers that I support have less than 2Gb of memory and they all perform just fine.
Most were upgraded from 32-bit SLES9 to 64-bit SLES10 SP2.
In my opinion Vista & 7 x64 are a tad snappier than x86 regardless of your RAM configuration. It's not much, just a general feeling of improved responsiveness. Obviously if you have 3GB of RAM or more, then 64 bit is a given.