Yes, what he said. But also, remember you are going to be sitting, and getting in and out of your vehicle.Aside from what type of gun or how tight your pants are, it also depends on what type of body you have. If you're lean and rock hard your body won't give much, but I'm well rounded and my spare tire has lots of give. I use the same pants but I loosen my belt one notch when carrying IWB.
You can test yourself easily by putting a holstered gun inside your pants, even if it's not an IWB holster. Walk around the house for a while and get a feel for what it's like carrying your gun in there, and you'll soon know if you need to go up a size.