I had a friend that felt the same way as you did. He told me when vaping in public he felt like an oddball and was further troubled by the fact about the controversy surrounding vaping. He began by vaping outdoors- in the backyard, next on the front porch, next in his car, next he'd have 1 last toot before he'd leave his car in a parking lot. He then began vaping in public away from people and now, 2 months later, has no problems vaping openly. He told me it was all about "baby steps" and convincing himself there was nothing wrong with what he was doing and that he wasn't an "oddball". The biggest breakthrough came when people actually asked him about the Ecig and were genuinely interested- many of those same people actually congratulated him on switching to vaping...he now looks forward to being approached in public.
Happy Vaping, Vapoor.
Edit: In regards to myself I never had that problem because I felt very strongly that it was all about my health. I've never had anyone approach me in a negative way about my vaping but have had lots of interest, positive input, and pats on the back.