Musings on the Next Search Engine
Google’s began by indexing massive portions of the internet, and rating sites according to which sites had the most links going towards them. You might state the general goal like this:
We will write a program to rank the quality of writing on websites.
This worked well for a while, but people with businesses saw how to game the system quite early, and generated websites which had many links pointing to their business. Google responded, and a cat-and-mouse game began, where Google developed more advanced ‘spiders’ to crawl across the web, sorting and rating the quality of websites according to rules which would not allow corporations to trick them, and the corporations developed better tricks, then eventually split off into specialized companies who had the sole job of ‘SEO’ - full time staff would attempt to trick the spiders.
They have been playing the game for some decades.
Somewhere off to the side, one of the weird techie sites had created an excellent ranking algorithm, but relied on humans to turn the wheels by voting. Reddit gained a reputation for excellent links, but poor search functions, so the web now has a happy-balance of Google indexing Reddit, and then showing people links to Reddit answers.
This seems like the best result, but we (‘humans’/ ’techies’/ ‘people-who-want-information’) could approach this more directly. We might abandon the notion of robots rating writing quality, and instead focus on hobbyists rating writing quality.
I say ‘hobbyists’, because we don’t have enough specialists to read all the information on the web. Specialists in any field probably use their time for research more than staring at Reddit. But hobbyists, almost by definition, have plenty of time.
For a charitable example of what I mean, I can tell you exactly how good someone’s writing about tabletop RPGs is, but couldn’t tell you about articles on gardening in Kenya, as I have no information.
A less charitable example might include discussions on Economics. Everyone seems to think they have something important to say on the topic, but I can’t see how much of these statements have any use to anyone, and I don’t even know how much Economists agree on their own internal disagreements. People think science has a lot of internal disagreement - and those people have no idea how uneducated they are.
But problems with the Dunning-Kruger effect and everyone having an opinion don’t worry me, because I didn’t come to propose a solution to a problem, I’m here to make a statement and a prediction.
The statement is that robots cannot rate the quality of writing well, and will continue to suck. The prediction is that anyone who cares about writing quality will move towards something like Reddit - where robots may appear, and perhaps suggest various articles, but only humans cast votes.