I'm getting confused by the answers, so let's try this.
There are two parts, the thing that goes from the wall outlet or cig lighter adapter to a USB female connector (like found on the side of a PC), and the thing that has USB male on one side, and your PV battery connector on the other. The first part is a USB power supply. The second part is a PV charger.
The USB power supply output should be equal to or greater than the PV charger input. Every PV charger I've heard of has its input and output numbers stamped on it, as do USB power supplies. For instance the PV charger may say "Input 5VDC 200 mA" (or 0.2 A), "Output 4.2VDC 150 mA" (or 0.15A).
For the above case, where the charger input is specified as 200 mA, you want to make sure your USB power supply is rated for 200 mA output or greater. A 500 mA, or 1000,or 2000 mA USB power supply is absolutely fine, but you should NOT be using a USB power supply that is only rated 100 mA. The PV charger wants to sink 200mA, and if the power supply cannot source that much it is being over-driven, out of spec.
Now for that PV charger. You do not want a PV charger with too high an output current for your particular PV. For instance a standard Riva or eGo charger may output 4.2VDC 500 mA. You should NOT use this charger on a small standard 510 battery, one that is only 180 mAh capacity; the charger is driving too much current for that little battery. The converse, however, using a little 4.2VDC 120 mA output standard 510 PV charger on a big eGo or Riva battery, is okay. It may take a long time to charge the battery, but the charger is not outputting too much current, it is safe to do.
The other, ugly, aspect of chargers is that some vendors use reverse polarity. Typically the center contact of a battery is the positive, +, "hot", while the outside edge of the connector is negative, -, "return". Some vendors instead have the + on the outside of the connector, with the - on the center post. This is just a flat-out ugly thing for any manufacturer to do, but it is done. For example, I believe the Blu is a 510 connector, but reverse polarity - you cannot mix and match blu PVs and chargers with other, standard, 510, equipment.
(atomizers and cartomizers don't care about polarity -- they are simple resistive items, doesn't matter what direction the current is flowing)