I'd have to say it's probably the menthol.
I use TFA Koolada, but only in fruit mixes. Found from experience it's absolutely vile in anything with cream or yoghurt flavour in. It's tastes, as you say, like moldy socks. I made a blueberry peach yoghurt with only 1% menthol once, tried after 2...