They're made up of several different things (polyester, steel, tungsten, etc.), none of which is bad for the environment. So, from that standpoint, they're not like the curly fluorescent light bulbs, which practically need a hazmat team to dispose of properly.
They're also little, bitty things that take up very little room in the landfill, which is another concern for rapidly filling landfills. They're not like fully-loaded disposable diapers.
Of course, recycling ordinances vary around the nation. But it seems to me that if you don't have to recycle used ballpoint pens, then you wouldn't have to recycle used cartos.