You might be out of your mind — but your ears don’t deceive you on this. The phone internals that process your music files and send out an audio signal are not optimal. Even going from an iPhone 5S to a 6S, there was a noticeable improvement. Still, phones are not focused on producing superior music sound quality.
Your USB drive is relying on the auto radio internals to do the processing and send the sound signal. The radio system is dedicated to audio processing.
Once I moved to iPods and the like, I researched and then bought a DAC. That’s a Digital-Analog converter. It is as small as a cell phone. It’s connected to the phone and the receiver. It keeps the phone’s conversion out of the process and handles it instead.
Huge difference! It brings the sound “forward” and adds a
sense of “air” or spaciousness. Compression in digital music files takes away those qualities. The DAC has superior clarity too.
It was a hassle to haul it and its cables around. But music in the car was always a big deal to me so it was worth it. On the shelf next to my receiver, it’s very handy. (I bought a Harman Kardon receiver that produces only clean sound. No digital processing.)