The man was panhandling/begging/soliciting money. In most communities there are ordinances against this. We have had people do this on occasion in our neighborhood and we always call the police who respond eventually and if the guy is still around they take him for a ride.
In Florida, when they cross your property line( front lawn/driveway) they can be charged with trespassing if they refuse to leave when asked. If they get surly with you there is always stand-your-ground. We as citizens don't have many rights anymore, but as long as I pay my taxes, my house is mine and my word in it is law!
They used to allow panhandling on street corners in my city. The place got over run with dead-beat types asking for handouts. They began to get pretty aggressive and made my wife extremely nervous to the point she avoided certain intersections. Most of them were homeless and started tent cities on city property and in parks where they used any convenient area to relieve themselves. The city finally got tired of all the complaints and made it illegal and began arresting those who still tried it. Practically over night they fled like locusts to another city that had no such ordinance.