Odds are nothing bad will happen, but at the same time there is a chance. It's always better to be safe than sorry.
Look at it like being bitten by a shark while swimming at the beach. A million might might not get bit but the poor guy that does is in for a heap of pain & suffering.
Best to keep an eye on the batteries but a once in a while, "opps, I forgot" shouldn't be a problem.