I think it really depends on what exactly you're wearing. I find that most people just assume you're in the military if you're just wearing digital camo like desert, cadpat or marpat. I find people start to stare with a little bit more concern when you wear woodland or urban. I've never seen anyone walk into a store/restaurant wearing all their gear, and I hope I never will. Thats where I see a real problem starting for obvious reasons.
"It's not the size of the dog in the fight, it's the size of the fight in the dog."