I’ve never heard of handbag used that way.
In the old days, we used to say pocketbook. Those were generally stuff, smaller accessories, often with a clasp rather than a zipper closure. A purse used to mean a very small change purse with a kiss clasp, used by women or men.
Later, people started calling all of them purses. Never cared for the phrase myself. Handbag is a current term for most styles whether they are meant to be carried by hand or not, which could be broken down into categories like crossbody, satchel, shoulder bag, totes, etc.
It’s just easier to have a generic term like bag or purse instead of expecting everyone to know what you mean when you use a more specific word. My husband knew I liked Dooney’s, but he couldn’t tell a bucket from a shopper.
