You said the Springfield XD4 is just too big to carry frequently:
Spr XD4 9mm -- 7.3" long, 5.5" high, 28oz.
In what way is it "too big"? Length of the gun, overall height, size of the butt sticking out, thickness, weight? More than one of these? Does it simply "stick out" too much (which might be the belt/holster or even your clothing, not the gun)?
Keep in mind that a couple of your other guns aren't really that different in size, least of all the SP101 (which many feel is reasonably compact):
Ruger SP101 -- 7" long, 5" high, 25oz.
SIG P239 9mm -- 6.6" long, 5.1" high, 25oz.
S&W 642 -- 6.4" long, 5" high, 15oz.
Beretta Tomcat 32ACP -- 4.9" long, 3.7" high, 15oz
Your possible new gun, along with a couple comparison examples:
Spr SC XD3 9mm -- 6.25" long, 4.75" high, 26oz.
Glock 19 9mm -- 6.9" long, 5" high, 21oz.
CZ 2075 RAMI 9mm -- 6.5" long, 4.7" high, 25oz.
Springfield EMP 1911 9mm -- 6.5" long, 5" long, 25oz.
Glock 26 9mm -- 6.3" long, 4.2" high, 20oz.
I'd say that you already have a couple of decent alternatives, and you're often carrying those. If you truly like the XD format, the subcompact version (3" bbl) will be nearly as small as the Glock 26, though similar in weight to the XD4 you already have. As suggested, if cost is the issue, you could sell one of the others and acquire the XD3.
What would I do? If I loved the reliability and format of the XD4, I'd simply acquire an XD3. That would give me two of the same platform to allow for different situations (temps, clothing). If I needed the cash, I'd sell either the Ruger SP101 or the SIG P239 to finance the new purchase, keeping the S&W 642 since it can go everywhere the larger SP101 can go and would offer me a decently compact choice when needed (though, .38 spl).