.380 would be as small as I would go.
My feeling is a .380 > nothing.
Most of the time the sight of a gun is going to stop someone. If not, then shot placement is key. I doubt a BG is going to stop and say "Please don't shoot... wait... Is that a .380? HA! I'm going to rob you anyway..."