The first part for me is about the huge clouds some people produce while vaping, do we know scientifically how quickly it dissipates? both outdoors and indoors? how much nicotine is present per cubic metre of air or the second hand effects on passers by? I don't think we do exactly, nor would a non vaping member of the public so he may be forgiven for being slightly offended by the possibility of breathing in the invisible contents of someone's exhalations, particularly when they seem to be partaking in some form of recreational smoking activity.
For the second part, I'm not afraid of confrontations with nonsmokers, i'd just prefer not to offend them. Whatever vaping rights we're fighting for or eventually end up winning - they will always be trumped by the general public's absolute right to breathe clean unpolluted air.
It's about looking at vaping from an outsider's perspective, to him it looks like you're smoking something, and even after the vapour clears he's still going to think its there, because that's just how people are and not all gases are visible. If some guy is out with his wife and children at a restaurant table i'm not going to start vaping right next to them, out of respect, not fear.