I've heard that small children/infants are sometimes sensitized to things by being exposed to them too early. Paradoxically, if you grew up in too sterile an environment you may not develop tolerances to things. I would keep it away from babies/infants, by vaping outside. As they grew older and their systems are more mature, they are larger and more tollerant, I'd try to "keep it down" but not worry about 2nd hand vape too much.
Who knows... this could end up being completely backwards... but it's what I would instinctively do. Someone that understands this stuff may be able to comment on how our bodies develop allergies, for example. As far as dangers of 2nd hand vaping, the data is incomplete, but probably much less of a problem than tobacco.
I've read many times that 90% of the nicotine is gone, etc. I seriously doubt that, but hey.... It's not very concentrated as 2nd hand vape (read very dilute). Also, open a window if you don't have pollen issues too.