I had a few small holes in my aluminum siding from where an old down spout was mounted. Filled them up with caulk and didn't think any more about them.
Woke up one morning with pecking noises that sounded like they were coming from the attic, so I crawled around up there looking for a trapped bird and couldn't find anything. The next morning, same thing, went up into the attic again, nothing up there.
The third morning I poked my head out the back door thinking maybe something was on the roof and found an American Goldfinch going to town having pecked out the caulking in those small holes. By the time I got back with the BB gun he was already gone. The next two mornings I got up and went and sat on the deck waiting for him. Got him with the BB gun both times. I was surprised he came back to get shot a second time, so the next morning I went out there with my 22 caliber pellet rifle intending to end it.
He thankfully didn't show up, but my regular Orkin man did to do his perimeter spray. He was understandably curious as to why I had a rifle with me, and said that the insect spray they use doesn't kill the birds, but it does repel them. He also recommended refilling the the little holes with a bit of caulk, then a piece of steel wool, then more caulk on top. I did that real quick and he started spraying the holes afterwards and I haven't had an issue since.
I'm not sure what insecticide Orkin uses for the perimeter spray, but that might be worth looking into to repel them.