If the hole moved at the speed of light, the electrons displaced have to move at the speed of light.
Electricity isn't really like most things we encounter. An analogy often applied to electron drift is that sound travels in air at... well the speed of sound, but that doesn't mean there's air moving at the speed of sound.
Here's another analogy.. Suppose you have a stiff rod that's 1 mile long floating in space. If you hit one end of it with a hammer, that compression wave is going to travel down it at the speed of sound in that material. However, no part of the rod itself is travelling at the speed of sound.
Or lets put it yet another way.. Electrons have mass. It's a small mass but it's a mass. Things with mass can't travel at speeds that are a significant fraction of the speed of light, because it would take infinite energy to accelerate them to there.
But there is a name for an electron flying near the speed of light... the Beta particle, which generally comes from radioactive materials where the high energies involved in nuclear decay can accelerate electrons that much.