It's been touched on some. But, I'll go ahead and post any way.
The most dangerous thing you can do is over handle your firearm.
Your already doing things that are a lot more dangerous than keeping a loaded gun safely tucked away in a good holster. (Loading, unloading and the like really adds to the danger of an ND.)
If you are truely concerned about safety, the BEST thing you can do is load up and keep your firearm in a proper holster until it's ready to be used. (Chambered round; or NO chambered round)
Don't mess with it unless you're practicing dry fire, shooting or cleaning. (Again, loading, unloading, chambering and re-chambering ALL up the anty when it come to the ND risk.)
But, if you use your PCW as your HDW and you have to take it out of your PCH and place it in a safe or other holster at night, just keep your trigger finger away from the trigger and you'll be fine.