I have this same approach. I vape whenever where ever, but would gladly stop if anyone were offended. When I'm indoors, on public property, I hold the vapor in long enough to let it dissipate.
I might stop if someone were offended, depending on where I am. If it's outside, no. I don't feel obligated to knuckle under to someone's bullying. If it's indoors, it depends on the environment. I think I'd tell them that I would stop of they could give me any rational reason for being offended. I'm sincerely interested in what it is that's offensive. If they can even remotely describe the odor and say it's offensive to them, I'll stop. If I'm inadvertently blowing clouds and it's distracting, as in a theater, I'll stop. However, if they say they are offended merely because it "looks" like I'm smoking, then sorry. That's not good enough. I'll be glad to politely explain that I'm not smoking, and the difference between vaping and smoking, but I don't live my life submitting myself to the whims of irrational people. Only those totally lacking in self-respect kowtow to the demands of irrational random strangers.
To those vapers who would submit to the demands of random strangers, I'd pose this question. If you were drinking a soda from a plastic cup and someone asked you to pour it out because it looked like you were drinking a rum and coke and they, as a staunch teetotaler, were offended, would you do it? Why not? They have the same right to be offended as someone who is offended by vaping because it "looks like" smoking.