A simple answer would be elitism. There's a widely held belief that if you vape, and didn't start as a means to quit smoking, then you're doing it because it looks cool - making you a poser.
Elitism or disapproval because you're 'not exactly like me'... These beliefs and behaviors are based on ignorance and/or prejudice. Which we all know is wrong. There are some legitimate concerns though, at least from their POV.
There's a lot of fear and anxiety in the vaping community surrounding government regulations. If some of these misinformed (or bought off) politicians got their way these regulations would shutter many of the non-Big Tobacco vendors, which would be a travesty for the vaping community.
Many of the attacks made by politicians and organizations claim that vaping entices your average VNS (vaping non-smoker) to start a detrimental behavior (oral fixation), and that many of the VNS start vaping nicotine when they otherwise wouldn't. This gives the attack groups a strategical advantage, especially when the 'do it for the kids' platitude is used as a rallying cry.
There are many questions surrounding the science of vaping; it's worlds better for a smoker to quit and to start vaping, but it's a step backwards for a non-smoker to start vaping.
Vaping ex-smokers tend to be older than vaping non-smokers, so there will be a disconnect between the two groups. Many of the former don't appreciate that vaping has been adopted by the 'hipster' community. It's merely a trendy hobby for the hipsters, not a life or death situation as it is for them.
Many ex-smokers believe that vaping wouldn't be as scrutinized nearly as much if vaping was only practiced as a means to quit smoking. IMO these people underestimate a) Big Tobacco and their lobbying arm, and b) politicians with their hands out.
I personally think the more people that vape the better. It gives us a larger community. More vaping sales, more competition, better technology, better prices. It gives us a larger voting bloc, which is crucial in the fight against lobbying resources. BUT...
The only thing I ask (and this goes for vapers of all backgrounds) is that they take it seriously. Vape responsibly and with decorum. Stay informed. Educate others. Join CASAA and similar advocacy groups, because there are people who are fighting to take our vaping rights away from all of us. This not only diminishes our individual rights as a whole, but is literally a life and death situation for millions of people.
As long as the above guidelines are followed then I could honestly gives two $&!#$ about why someone started vaping.
TL;DR Many vaping ex-smokers believe that non-smokers who vape only bring negative attention to the vaping community, resulting in a poor public image, exorbitant taxes, and stifling regulation.