Beat the peewaddin out of those magnets but they just wouldn't break pressed into those bores. Got the elcheapo drill press out and no way a drill bit would do the job. Ruined a diamond abrasive bit trying that. On a hunch I chucked up a small triangle file and after some consternation and cooling sessions got
through the magnets with a small hole and into the aluminum.
Tried pounding them again thinking the hole would give some room for breakage but no luck. Broke off the end of the file and did it again.
Still couldn't break those things up so I broke off more of the tapered end and did it again. This time the hole was big enough that the edge of the magnets crumbled with the drilling without the file hitting the edge of the holes.
Took 3 hours total of drill some, cool some, repeat and sacrificing some tools but the small triangle file is the way to go. Got the calipers out and I have magnets close to the proper size but not quite so time to get some magnets and clean up the top cap in the mean time.