I don't want to overwhelm you, but there's a relationship between the voltage you use and the resistance of your coils to how long your battery will last...it's not just voltage. For example, lower voltage with higher resistance will draw more power (amps) from your battery than, for example, lower voltage with lower resistance. It's not super important, but it has an impact on your battery life.
4 hours seems a bit low for usage time to me, but the charge times seem right. 2-3 hours is typical for me, on a computer (using .5mA) or a wall wort (using .5MA). Some USB ports provide less power, though, so that would increase your charge times. I don't recommend using a USB hub as these almost always reduce your power input significantly. Laptops also usually feature reduced power outputs to help with battery life.
Batteries do have a memory, too. It's best to drain it all the way down and then recharge. Topping off, as it were, over time will affect the batteries understanding of where it's top and lowest levels are. I'm inclined to think, though, that you might have a bad (or a better term, "less good") battery. If possible, I'd try to exchange it.
What I really wanted to say is...why don't you have two batteries? Charge one, vape one? Or three? Charge, charge, vape. Or seven, like my foolish self? Charge, charge, charge, spare, spare, spare, vape. I don't have battery problems, I've found.
Good luck figuring things out!