The short version is that you can only draw when you would be justified in shooting (although some states do recognize that drawing to deter an attack before the lethal threat is upon you is not actually brandishing; you need to check on how this applies in your locality). Drawing without cause or to recklessly intimidate is the crime of brandishing.
A credible lethal threat faces you if the potential attacker has the Ability to do you harm (e.g., a weapon), the Opportunity to do you harm (e.g., a guy with a knife who is close enough to use it on you---look up Tueller Drill for some info on this), and by his behavior or actions is putting you in Jeopardy because of that Ability and Opportunity. Together these are referred to as the A-O-J criteria. (Important example: every time you are at the shooting range, everybody around you has the Ability and Opportunity to do you harm, but you take no action against them because there is no Jeopardy.)
The situation you are describing is covered by the legal concept of disparity of force, that is, the notion that an opponent or opponents may have the Ability and Opportunity to do you lethal harm even if no weapon is apparent.
Now, the law (the case law, essentially) enshrines some examples of disparity of force: multiple attackers (though generally more than 2-to-1 for an obvious disparity of force to kick in), disabled victim and able-bodied attacker (also temporarily disabled, like the fight victim who ends up on the ground and then the attacker presses the attack), and a few others.
To get back to your original question, there is no hard and fast legal precedent based on size and strength of an individual unarmed attacker. Generally a woman would get more lenience in court in applying lethal force against an overpowering, but this is not the same as saying you have the right to enforce a 15' exclusion zone around your person any time you wish.
If anything because you may have a lower threshold for legally justifiable use of force, you need to be more vigilant about avoiding the situation that might make you need to use it.