As you might imagine, speed is only one component of an effective defense. While important, there are many other crucial factors. In time, with competent training, you should certainly be able to be "fast enough" with whatever your selected platform is. It'll just come down to the mix of criteria you want to have in your defensive situation: how much training investment; how much time focused on these things; what sort of gear; what "power" of cartridge, capacity, etc.
For me, the mix of higher capacity, lighter weight, smaller size and comparatively faster follow-up shots have drawn me to the 9mm platform. Of course, having been almost exclusively on that platform for 20+ years, it turns out that's what I shoot best; though, I suspect that's as much due to simple familiarity as anything else. Still, your point is a good one. Different calibers/power can affect people differently. You might well find the mix of pros/cons works better for you in a different caliber/size of gun. Up to you, but many have made a switch for those sort of reasons. IMO, they're real, and they'll exist so long as you haven't been able to equalize the differences through training.