The USB chargers will take longer because they output less power, but it's handy to have one as a backup.
I think it's a little more complicated than that.
USB 2.0 specifies that unless the connected device identifies itself as a high-power device, that it will automatically be treated as a low power device with a max allowed current draw of 100mA (1-unit).
So even though a USB 2.0 port is capable of 500mA (5-units), unless there is active communication between the device and the host bus, then the full current capability will not be available.
Now, there is an expection to this though. If you are using a powered USB hub, then it treats all devices as high-power and will allow the full current amount available to any device.
When you plug the same USB charger into a 300mA wall adapter for example, then the full 300mA is available to the charging circuit. The wall adapter is a passive device that doesn't know or care about what is attached to it.