I happy the mods gave us a heads up this time and we can work a transition to the new threads. I'm not so happy that even though the old thread will be around - we'll really lose a lot of history as it won't be so easy to go back in time to older posts and new folks joining won't see the long trail we've been on.
Me thinks (OK I spent 45 years in the computer industry so I have an opinion). There's got to be a flaw in the forums code if it can't handle large threads. There's things called databases and indexes into the database to optimize and avoid long access times to a particular record (post). Me thinks they are taking the easy way out - eliminate big tables (threads) to search in the database. But the admins/mods of the forum aren't the designers/coders of the forum code, so it may be out of their hands and they are doing the best they can with what the have.
I do remember when I first got on the World Wide Web. It was totally text based in the beginning. I was truly amazed I could access libraries of information all around the world from my Commodore 64. Remember "Gopher". It was the 1st search engine to allow looking for stuff on the World Wide Web.
FIDO on a Vic-20, at 150 bps... almost as fast as I could type
Not all index algorithms are equal, but I think your point about the Sysops is more to the point - they're not the programmers, and so are as stuck as the rest of us. And, of course, the larger the database, the longer it takes to search the index. Carridwen did mention "performance" issues, iirc.