So far the OP has performed in a reasonable manner. There are other people nearby so it wouldn't hurt to ask those nearby to call 911, which might cause the suspicious man to think about going somewhere else unless he is a nut-case or doped up.
With no verbal threat having been made, and no weapon displayed, evasion tactics remain the best option. If the guy is able to make direct contact with you the only remaining option is physical resistance. If you are real lucky others in the area will step in to assist you. If not, you will be forced to defend yourself to the best of your ability. This will include the need to safeguard your weapon.