Ok, this is incredible. "Holster wear" is a normal by-product of carrying a firearm either concealed or openly. No product or method of carry will prevent wear. Your carry weapon is a tool and that tool (though I like mine to look nice) is just a tool like my screwdriver or crescent wrench. Leather, kydex, nylon, whatever, its all going to wear. Some more than others. Someone said that leather will wear faster, again, this depends on the leather. A custom made leather holster will NOT wear as fast as a mass produced leather holster because of tolerances. The tighter the holster, the less it wears. Which brings me to the drawing/reholstering or normal EDC wear. Drawing will of course wear faster than just letting it sit in a holster. But not doing your practice draws concerns me more than the wear on your firearm. Movement of your firearm inside of the holster during normal EDC will wear but at what speed is determined by the holster, again if its custom or mass produced, nylon, or kydex.
All in all, the wear on your finish is NORMAL and should not be looked at as a problem but more as your personal "touch" that makes your firearm "yours."