Yep, I toss them when they won't hold a full charge or notice a significant performance drop.
I don't abuse my batteries, I stay within CDR, charge them when they drop to around 3.6V and rotate them. I've got batteries over 2 years old and still perform well. It's all about taking care of...