We have similar ground living critters, and yep, they can be nasty little buggers. If they'd stay out of garden areas I'd leave them alone...
But when they do, well, get a good wasp spray, wait until dark so they're all home, and soak it down well. Usually works on the first try, but if the nest is too deep it may need another dose.
Barring that, I've read that a couple teaspoons of dish soap in a water sprayer can work, but I've not tried that.
It may also be possible to encourage them to move, by REALLY soaking the nest with the garden hose. Yeah, right, I'm going to stand near a wasp nest and hose it down.