I've been wearing contacts for 24 years, without a single problem that had anything to do with the contacts. For me, too many things can, and have, happen with glasses and if you don't have a backup pair on you when that happens, you're SOL. My vision is 20/15 (better than normal) with my contacts, 20/20 with my glasses and 20/400 (probably worse now) without them. Even if I lose my a contact, I still have the other one. I've had glasses where the frame literally broke in half while I was cleaning the lenses, I've had nose pads fall off, temples fall off, lenses pop out...I wear my glasses around the house, but rarely when I leave.
For the problem described, the simplest solution aside from the XS sights would seem to be a laser. Once it's properly zeroed, you can focus on your target and look where the dot is; that elminates the problem of the front sight being blurry.
Now, granted, Mr Murphy can come along and knock your laser out when you need it the most, so you do need to at least be able to hit the target at self defense distances without it, but point shooting can get you there.