His well-documented taste for the trappings of monarchy may have had a little something to do with it as well.
Part of it, but the Sedition acts were the prime mover there. Ran right into the face of freedom of speech and press and the people really didn't like that much. Still it was a close election - the last 'close election' though, since the Republicans dominated the '2nd Revolution' for 24 years of Jefferson, Madison and Monroe - all two termers.