Welcome to ECF, JHoun...
Generically speaking - button "wobble" (or angular tilting within the lateral axis) is typically due to an excess of radial/lateral clearance (much like a piston in a cylinder). You have two basic 'movements' in a
mod button - axial, or the lengthwise button 'press'... then the radial/lateral fit between button/shaft and outer switch assembly housing, which is the source of wobble.
I doubt you have access to welding equipment or a lathe, so we'll cover the quick and dirty, 'scissors and ruler' method. ;-)
To reduce the latter form of clearance - in effect reducing wobble, you can shim the button, shaft or both. This can be done with anything from actual shim stock, made of SS, brass, aluminum or copper... to strips of beer and soda can.
Although it's unlikely you have a dial or digital caliper, with one you can measure your clearance accurately. If not, you have to test fit various thicknesses until you've taken up a good amount of play,
without button binding in the switch housing.
An example would be - if you have a radial clearance of say, 0.006". A strip of shim stock 0.002" thick and wrapped around the button or shaft will reduce that radial clearance to 0.002", in effect cutting your wobble down by approximately two thirds.
As nothing you're doing will be no more permanent than the life span of the shim material used, and nothing is being permanently bonded in place... all is reversible to the mods original condition.