I think, Jon, it's easy for people that have had lot's of training to forget how little training most gun owners have had, and how little they shoot, and perhaps even more significant that few ever experience the intensity and fear of a gunfight.
We can learn the home layout, but we cannot force an intruder to pose at a safe location or angle for us to shoot him. Due to the location of the threat, the location of loved ones, we will likely not be able to position ourselves in those 'safe' scenarios.
I agree with everything you say for a perfect world, i.e. that all gun owners would be great shots, train under stress, etc., but the reality is they aren't and don't do any of that.
As for the reason some choose birdshot, most do so thinking it's less likely to penetrate walls with enough energy to be a risk, more than recoil concerns. Many people have close neighbors with thin interior walls separating them, so there is a concern, but unfortunately birdshot does not eliminate the threat of wall penetration. As evident from the video I posted, birdshot can easily penetrate two pieces of sheet rock and blow out both sides of a pumpkin and blast the insides all over the wall.
There is a also the mistaken idea that birdshot is a reduced load, and it can be, but 00 reduced recoil may not have as much recoil as some birdshot loads - it just depends on the specifics of the birdshot. But the physics remain, larger pellets retain more energy, travel further, and penetrate deeper than smaller pellets. Unfortunately, both birdshot and reduced recoil 00 can penetrate walls, at close ranges, and seriously injury someone.