I felt the same way when I first started EDC'ing a couple of years ago. I think it's all about finding a carry mode and clothing you feel comfortable with and eventually you'll get used to it. One common piece of advice is carry at home for a while. It gets you used to carrying and not worrying about it and you can check yourself out in the mirror occasionally to see what others see.
I pocket carry an LCR snubby. I have never had anyone notice it and although I know it's there, I don't notice it either. There is a bulge in my pocket (see post #2 for the obligatory joke), but visually it could just as easily be a fat wallet. If I am carrying anything larger, which is rare, I will just open carry my 1911 with my snubby as a backup. It's funny, I haven't had anyone notice that I'm open carrying either, or if they did, they didn't react.