The "stigma" probably has to do with the fact that this is concealed carry. People don't see it, so they assume you're unarmed. They'll only hear the stories of BGs doing bad stuff with guns.
I believe to get people to ease up, more open carry would help. Then people could see non-LEOs or BGs living their lives like everyone else.