I wouldn't call it insane. A second-best possibility, yes. But better to have it in an awkward position than not have it at all.
There are a variety of man-packs around, small and medium. Would you consider one? Either for the gun, or for everything else, leaving your pocket(s) empty for the gun?