IMO.... People who think drawing a pistol simply due to a panhandler coming too close is appropriate need to have other options. I'm not sure how comfortable the OP is with going "hands on" with someone, but drawing down on a panhandler because he got too close is an iffy response at best. Some people are "close talkers", and some people just try to be intimidating.
As far as how "approachable" you are, as stated already, I dont think it matters much. Panhandlers work on percentages. If one out of 10 people donate, they best be on their A-game if they want to upgrade from domestic to imported beer.
Or, just get a barb wire tatoo around your neck