I'm the same way. I just avoid doing it anywhere that is going to stir questions. I work as a consultant at many clients and I just take it outside to avoid confusion and attention. When I walk around downtown, I just kinda avoid eye contact with people who look like they are going to want to bum because I get tired of explaining why I don't have a cigarette or a lighter for them.
Ironically, after switching to e-cigs, I'm at most freedom at home alone.
vaping in public is what has become more tense. At least I can vape in my apartment now. That's the best freedom.