Golden Rule, man!
Wouldn't you have liked it if the drunks had stopped screaming? Don't you wish the drunk loud people were "bothered" enough to shut up?
The people angry at you vaping next to them indoors feel the same way. You should be bothered because it bothered them and decent people care about how they are treating others.
Again, nobody has ever seemed the slightest irritated by me vaping, but often curious. Yes I would have appreciated if those people "shut up". But if they just kept their drinking subtle and conversed at a reasonable volume for the situation, it would've been enough.
I'm always considerate of others when I vape in public, but the possibility that someone might get offended will not stop me entirely.