That regulator is a POS, works more like a resistor as it does not maintain a nice constant 5V eventually dropping into the 3V range, and issues have been reported when using a Joye atty (cutting out). And seeing as the OP has no master switch that regulator with the control pin will always be using a small amount of current, until he pulls the batteries.Don't change the switch, change the reg.
Get the 4 pin reg from madvapes.
With the appropriate switch the OP has, this is a much better solution.
Digi-Key - 296-11414-5-ND (Manufacturer - UCC283T-5)
The whole point of building a 5V regulated mod is to have a consistent 5V the whole time.