Personally I found 3" guns, especially the .40 cal. to be uncomfortable to shoot or carry. As far as carry the short barrel and heavy grip seems to always be pulling the holster into my side.
I found the short barrel combine with stout defense load to be a snappy recoil. Also the 3": barrel didn't seem to be as accurate for me. I traded to a 4" 1911 SA champ and found much improvement in overall accuracy and comfort. (I still use a full size .40 for winter carry ).
I don't think a 4" XD is all that big to concealed carry, but a lot depends on clothing, body type and holster.