I have noticed that it is more attitude and ones actions that makes folks feak out more than the attire. If I walk abrubtly into a store looking ticked off at the world, mumbling to myself, and just plain acting rude..that will cause folks to freak out more than if I walked into a store, held the door open for the lady behind me, said excuse me for bumping into someone, and picked up a childs pacifier that fell for a mom. I could be wearing the exact same clothes and looking scruffy.
I have told the tale of when I met a mom hiking with her kids. She obviously ws weary when she saw the gun. I was polite, talked to the kids when they asked me if they could pet my dog, looked the mom in the eyes and smiled to let her know it is OK.
Of course appearance is the first thing people notice. But attitude and demeanor is what will most likely put them at ease or make them freak out.