I do the same as Kembial (except with a tissue) but only once in awhile. I clean the battery end of a cart to make sure it doesn't get on the battery, so the bats stay very clean.
I imagine if someone is in the habit of sticking a battery without a carto into their pocket or floating at the bottom of a purse, then it could pick up some dirt/dust/lint/etc. and might need more cleaning, but mine is either in the pouch or the pcc when I'm out, or in a cup on my desk when not being used.