I've never been hot-headed or taken particular offense to anything anyone says to me. Almost to a fault. But even unarmed I don't see the point of responding to derogatory remarks... that is, after all, what the "bad guys" want. They want a reaction. That's why they are saying what they say. They want it to tick you off and they want you to respond and "disrespect" them so they have an excuse to escalate the situation.
Why give them the satisfaction? Even if it turns out that you win the fight you still stooped to their level by giving them the fight they were looking for.
If they want to bring the fight to me... okay.. let's go, but they are going to have to bring it ALL THE WAY.. I'm not going to egg em' on or give them even the most temporary of satisfaction in knowing they've bugged me.