For high frequency switching applications yes, gate charge comes into play and you need to minimize gate loading. For user input, not really. The time constant formed by the gate capacitance and even a 47k resistor is still very small. Plus you have to look at overall input impedance. It's not just the resistance of the pull down or pull up the gate charge is working against, there's also the impedance when the switch is pushed, which is usually pretty low.
With reasonably valued components, there should not be a concern over gate drive impedance for user input. Worst case, it will cause a slight ramp up or down, but nothing that would be detectable empirically. For very high current applications, you may have to consider FET heating, but I don't think that's an issue here. Surge currents have to be pretty large with power FETs before they become a consideration.
Often when accepting user input from a switch, you use a debouncing circuit that adds quite a bit of capacitance to the driving end of the switch. I set up my switch debouncers with 30k resistors and 1uF caps. Still not an empirically detectable delay and that's a time constant orders of magnitude larger than the one formed by a pull up/down resistor and gate charge of a FET. Though on an oscilloscope, it's quite detectable.
