Get all of it out. Powder coat stuck to the sides can get scraped by the spring when you reinsert it and wind up under the spring.
I cut a piece of green scrub pad, rolled it VERY tightly, and twisted it inside. Steel wool would probably work better but I didn't have any.
Even after that the...