Your control of the situation.....is that how they see you or how you see them? Or is it that you're confident and they can smell that? Kind of like a bee will sting you or go for the next clover.
Living in a college town there are tons of vagrants roaming around asking for money and I've had two yell and rant at me. Before carrying I felt kind of helpless, I would tell them I didn't have spare cash on me and just say sorry until they left. Now I see them walking up to me, I say NO. and they just leave me alone. I have more control of the situation.