You are anticipating recoil. After 20 rounds or so you are accustomed to it and overcome the urge to push the muzzle down.
Practice...practice...practice...maybe have someone else load your weapon (or not). When they hand it to you it may have a round in the chamber or not. The first time you light one off with an empty chamber you'll see pretty quickly what you're doing.