Everything depends. First, it's very important to use only a quality battery charger. The brands we usually recommend are
xtar, Nitecore and the Efest LUC line. There are a number of others that are just as good and often have extra features, so if yours isn't one of those let us know what it is and we'll tell you what we know about it.
If your charger uses a USB, you may be able to use it with your phone adapter (technically, it's an adapter, not a charger, but I admit it's usually called a charger) but it will depend on the output of the phone adapter. Do you have an amp or watt rating for it? It may or may not be powerful enough for the charger, depending on how what the charger needs.