Good question, I'd do a little experimenting with some of the crystals and different adhesives on a scrap of fabric.
With some of the pv cases I've done, some adhesives have reacted differently say clouding the backing of the stone, so that's also something to keep in mind. I'm also wondering about fabric glue, maybe worth a shot, HTH
Also wanted to add, that a damp q-tip does fast work of picking up the stones