Great idea with the center punch to tighten the threads back up JeremyR.
As far as a replacement "tactile" switch goes, I don't think that reducing the size of the switch is going to have any effect on how well it functions. As a matter of fact, I'm afraid that if you try to reduce the size, it may actually be more critical that you press the outer button in the center to make sure the switch below is pressed properly and that it fires. My suggestion is that if you plan to replace the original switch with a new one, use a switch that is basically the same size and that the legs that need to be soldered are in the correct spots. Having to solder a replacement switch to the board when the solder pads on the board do not match the legs of the switch will be difficult if not impossible. That's why when I purchased a replacement, I made sure to measure the distance between the legs on the original switch to ensure that the replacement would be a good fit for soldering.
Another aspect to take into account is the height of the switch. If the replacement switch is too tall, the board won't fit in the tube. If it's too short, your outer plastic button will flop around. The replacement switch I purchased on Digikey had the right dimensions for soldering the legs, but the height was a little short (P/N below). I used hot glue to build up the underside of the outer plastic switch and then cut the excess off flat with a razor blade so it would meet the replacement switch flush and with minimal wobble (if any).
The measurements of the solder contact spacing for the soldering pads are 5mm between the legs. The switch body dimensions are 12mm long x 12mm wide and 5mm tall. The switch is a Single Pole, Single Throw (SPST) "tactile" switch. You can do a search on Digikey or whatever electronics vendor website you prefer for a switch that matches these dimensions and it should work fine.
If you decide to search the web for a replacement, when looking at switch characteristics you may also want to take a look at the "durability" or "duty cycle" (number of button presses its lifespan is rated for) as well as if the switch is sealed or not to prevent internal contamination from e-juice, etc. The one I bought on Digikey is part number
450-2051-1-ND. That switch's datasheet reported the durability to 1,000,000 cycles (key presses). It was also the only button that was "sealed" against foreign contamination with dimensions that were close to the original switch. There may be other tactile switches for sale on Digikey since I last did a search that are better. The only thing about the above button that may not be ideal is that it isn't as "clicky" as the original switch was and it's slightly easier to press. I haven't had any problems with it, but others may have trouble getting used to the feel of it. Based on how they put it in their pocket, etc. it may fire accidentally more easily than the original switch. For me, it gives a solid "thunk" for feedback when I press it so I know it's pressed. I've used it in two different Sigelei V3's as well as a Vamo V3 with no issues and I expect they will work as long as the PV holds together.
My original write-up on how to replace the switch appears earlier in this thread
here if anyone is interested.