I assume you are referring to homeless people. Many, in fact I think the majority of homeless people do have criminal records and a majority of those have mental problems and can become violent with little or no provocation.
I would check with your local police department to see if they have statistics on the local homeless population and bring that to the attention of your employers. Voice your concerns over your and your fellow coworkers safety and make it a point of writing all this down with time, date and content of the conversations along with who you spoke to. Make a paper trail that can be brought to court if something happens and there is a criminal or civil action that results.