A couple of people here have got it exactly right already, but I'll recap.
Every time you fire a gun without hearing protection you do a small amount of permanent damage to your hearing. The damage done by a couple of shots is so minuscule that you will never notice it. Since none of us are very likely to be in long-lasting gunfights week after week, we really don't need to worry about hearing protection in this situation.
Going to the shooting range, which we should all be doing on a regular basis, is a completely different matter! In that case we are firing round after round and, as I said, doing it on a regular basis. The cumulative effects of not wearing hearing protection in this situation absolutely WILL reduce your hearing in time!
Beyond that, it is entirely possible that you will experience a temporary deafness after firing a few shots without hearing protection. This is a matter of perception (just like audio exclusion) and is simply the brain's efforts to blot out an excessive distraction. Unless the noise is off the scale (someone mentioned a grenade as an example) your hearing will come back relatively quickly.