While a shotgun has more stopping power, it's not as easy to keep it close to your bed when you're sleeping compared to a handgun. Also, maneuvering corners is more difficult with a shotgun and you can't hold a flashlight or open doors as easily with a shotgun.
These lead me to think a handgun is probably a better option.
That being said; handguns seem to make my ears ring more when not wearing ear protection...shotguns don't seem to bother my ears as much. Some people say that carpeting, furniture, etc in houses absorb a lot of sound when a shot is fired. Since I have never fired in a home, I'm not certain how much this helps...thoughts?