Found this while I was just looking around, kind of interesting...
http://www.city.nagano.nagano.jp/ikka/k ... aijin.html
That's not to say the word gaijin can in no way be used in a foul manner, but that definitely makes sense why a lot of people may feel that gaijin doesn't have a very nice ring to it. Especially in places like America, where you have so much diversity, it's kind of rude to just lump everyone that's non America and just label them as foreigners, even if that may be true.
From my own personal experience, I never really had much of a problem with being called a gaijin, as most of the time it's not like the person meant it in a really bad way or anything, and those that did use gaijin in a bad way towards me, really, I could care less about what those ignorant people have to say.
By the way, did you know that...
18782 + 18782 = 37564
Funny how that works out though, really made me laugh the first time I saw it.