I find that as long as you're not using the white with orange/red LED unit, you shouldn't get much attention or difficulty.
My unit is a Silver 4081 with a Green LED (standard Cigarette sized unit), and although I've gotten some odd looks, I've vaped in Walmart, Sears, Best Buy, several bars, my office, three different grocery stores, many gas stations, and the list goes on. I've only had one or two people actually ask me what it was. I also don't call it an E-cig, I call it a Personal Vaporizer, it is just a little less "Run for the hills, it's cancer coming" than calling it an Electronic Cigarette.
I do, however, make it a point to hold the vapor in a little longer so I'm not bellowing out plumes of vapor everywhere. Most of the time when I finally exhale, you can barely see any at all, if that. I think that calms the "run for the hills" types, when they don't see anything coming out.