First off, concealing a weapon under a layer or two of clothing will always present a slight hurdle when you draw your weapon (even while your weapon is concealed under a t-shirt), as you probably know....the only thing you can do is constantly train to access your gun in a expeditious manner while concealed. Do your drawing from concealment training both at home and the range (if permitted).
Everyone these days wears reasonably baggy clothes and untucked shirts. Depending on where you are at and the circumstance, fanny packs and vests don't necessarily give away that you are concealing a weapon. If you look out of place, then you are going to get noticed regardless of where you are at....doesn't necessarily mean that you are "made" as a gun carrier though.
For me, patterned untucked shirts help hide any larger weapons I carry like my duty SIGARMS P229 or my off-duty sig pro SP2340. If the weather is cooler, then a jacket or vest is obviously no problem.
I like to carry Kahr 9mm's off-duty so slightly oversized t-shirts work perfectly due to the small size and flatness of Kahr pistols. Larger guns do not allow me to carry them under a t-shirt; they will print nearly everytime.
I don't know the brandishing laws in CA, but I would imagine that if someone that says "baaaa" spots your gun and reports you to LE, minus any intent on your part to purposely expose your weapon, you will at least be confronted by LE, possibly have your weapon seized at the scene, along with your ID (and CCW permit), possibly handcuffed while they verify who you are and then release you, returning all of your property including your gun which may have been unloaded on the scene, and sent on your way.....with the LEO's possibly giving you a lecture on "next time" securing your weapon in a better fashion as not to frighten the horses. At worse, the responding LE agency could recommend you lose your license in the event you brandish your weapon even accidently, arrest and charge you but I would think that's a worst case scenario.
I wouldn't worry about "going baggy"....everyone does it; heck even fortysomething year-old old fart me and I get away with it everytime.