If you happen to be, like me, in the throes of hoisting a company that incorporates some flavor of Search technology as a key capability, you know that its value in managing and sorting the torrent of internet information pouring out of blogs and everything else these days is essential to the success of the business. This is true for Google, Technorati, YahoO!, et al as it is for any content-oriented business. With the ever increasing flow of noise out there it’s harder to find the signal: When I search for Vanilla Ice Cream why do I stumble over Vanilla Ninja? The real problem though, is this: although having an effective matching engine is critical to the success of the business, search is not in and of itself all that interesting. Google was probably the last company that made search itself interesting as an end-user value proposition — and as we all know, what really made Google interesting in the long run was what they did with users (and avertisers) once they had ’em hooked.

These days, solving the search problem is just one step on a long path to building valuable services that people enjoy and make use of every day. Deep nerds like tackling these issues because they have all the hallmarks of geek chic. They are difficult algorithmically, require massive planning from a scaling perspective, and require constant tweaking. Google was successful at attracting people to its search engine for two reasons: it had a cleaner interface (they hadn’t decided to become a Portal) and it had more accurate results (the other engines had become gamed). I’m sure Google has tons of patents around their search capability however I am too lazy to search for them because sifting through the results separating wheat from chaff would take way too much time.

And that, dear reader, is precisely the point. Google, too, has been gamed — as will every search engine that comes into common use. So what am I on about? Well, Jimmy Wales wants to open-source the search engine… and for the record I think it’s a great idea, and one that threatens GoOgle substantially.

My logic is this: If the value of a search engine is no longer the search engine itself, but instead the application to which it is applied, then why not accept its value as a generic must-have and open-source the thing? We can all benefit from the assumption that the search engine itself will always be gamed by spammers and sploggers and search engine marketers. Once we do that, creating a community that is invested in the efficacy of the search engine (because they’re making money from it) also creates a system by which that community is incentivized to keep the thing working properly as it’s gamed by persistent SEO gremlins. This is far more effectively done by a collective of companies than it is by a bunch of companies tweaking their own engines independently, pursuing near-term, interim, proximate advantages.

Wikia Search is nudging closer to existence, but I think it’s applying the brute-force labour at the wrong end. As Jimmy Wales becomes more and more assertive and aggressive with his crusade to fix search using an army of lemmings using the Wikipedia recipe, which means he’ll use the community at large to determine the merit of matches found by his search engine, he’s extending the Wikipedia model to searching. Users will “vote” matches to the top of the rankings.

interesting notion, but I think he has the right idea but might be missing the mark on execution. As anyone who’s watched Sanjaya on American Idol can attest, user-voting is not always the most expeditious method of ensuring quality. Wikipedia uses a broken-source (have I just coined this term?) publishing model: it achieves one thing very well (aggregating information and content from diverse sources) at the expense of the other (ensuring that information is trustworthy, balanced or factually correct is problematic). Applying this model for Search is therefore solving the easy problem (search engines already aggregate and index things quite well) with the wrong method (envision Wikia Search gaming teams in Bangladesh sweat shops “voting up” rankings for their customers on the engine).

So, right idea — wrong solution. Let’s create an engine that everyone can (and does) use, that everyone can tweak and repair, and that is policed by a foundation which has as its only goal the efficacy of the product. The Deep Thinker Nerds who like to fiddle with these kinds of problems will be attracted naturally to the project, and their incomes could easily be supported by the companies benefiting from the expansion of the technology. Jimmy’s in a position to lead this, to some degree, but he doesn’t evidently understand that the strength of Wikipedia will be the achilles heel of this project. He has claimed the high ground but I fear that he will inevitably fail.

-Ian.