He's your friend, your buddy.....and he knows you carry. Sense he dosn't carry it's something he'd probley look at/for every time ya'll are around each other. Him noticing you wern't carrying (and voicing it) was just part of ya'lls dynamic...wether you knew it or not.
Our friends and family that KNOW we carry watch us.....ALL the time. Be them for, aginst, or completly neutral, they will notice. We (as gun 'carriers') will always be watched by those that know we do. It's Nothing bad (most of the time), we're just 'diffrent' to those who see carrying guns as 'forgien' to them.
Very few of my family and friends know I carry, but those that do watch for me packing....or not carrying. My wife and (older) children see my carrying as a comfort to thier well-being more often than not. My mom, however, knows I'm competent and 'trained' but still has that 'cringe' that her 'baby' packs a gun......the whole "live by the gun, die by the gun" fear. That's not the case, but that's just her. Most of my buddies that know I carry pack themselves and we all look for each other carrying, just a natural observation between friends. Many acquaintances of mine have NO knowledge of my carry habits and thus don't even know to look OR what even to look for.
And as for confidence in your 'concealment', ANYONE who knows will also know what to look at/for to see that your carrying or not. Although most people you come into contact with neither know nor care, and won't spend the energy checking to see if the strangers around them are packing, NEVER believe that your sidearm is ALWAYS 'invisible'.
