We don't get that many complaints about the server speed, now. We recently changed up to a multi-server setup with a fast server app called Nginx instead of the usual Apache.
However there is a thing called a 'cron job' that hits big forum sites hard, it's where the software periodically has to update the database by running a rebuild of part of the DB. These cron jobs occur through the day, every hour for smaller ones and every 4 hours or so for major ones. A cron job hits the server hard and uses a lot of resources. If you find the site slow then maybe if you wait 20 minutes and try again, if the site is then at full speed, it was a cron job dragging it down.
Servers are expensive and we tend to rent enough for full speed 95% of the time, as renting enough servers to cover all cron job dragdowns would cost a bundle more. Nevertheless you should not see a major speed reduction as the biggest crons are run (or should be run) at 5am or thereabouts, where possible.
I think that in most cases, some kind of local issues are the reason. The first thing I'd do is change the DNS server IPs on your router, switch them to OpenDNS, GoogleDNS or ComodoDNS for example. You go into your router's maintenance admin panel (type ex: 192.168.0.1 into your browser address bar) and remove your ISP's DNS server IPs and substitute a better set such as OpenDNS. Fill all three slots, don't leave the third one empty, you can repeat one of the first two in the third slot, or use one from another service. If you leave the third slot empty, some ISPs will remotely fill it with their own DNS IP and use it in preference to your first and second choice. They can be tricky b's
The other thing is that all ISPs have a contention ratio, in other words they don't have enough equipment to service all their customers if they all came online at the same time. The contention ratio is commonly 20:1 ("twenty to one") or even 50:1, meaning they only have equipment for one in twenty of their customers (read the small print!) and the obvious result of this is at peak times such as 1pm US East Coast, especially if there is a plant issue somewhere, service will be slow.
One way to isolate local issues is to ask someone on the other side of the country if they are getting slow speed. Usually, they aren't, it's you. If it's definitely us, as several people in widely different locations and using different ISPs are having an issue, then we'd like to know the exact time and what timezone, please, so we can look at the logs and see what was going on.