Despite the fact that I often find myself on the opposing end of the table on most of what Microsoft does, I was really hoping to be able to agree with Ballmer on his assertions regarding Microsoft’s rejuvenated focus on search as quoted in today’s Financial Times article. I was hoping that, on the heels of their disastrously failed hostile takeover effort of Yahoo! that MSFT had a plan for Search that extended beyond paying people to use its engine, which has led to some amusing arbitrage opportunities reminiscent of late bubble-era scams.

Of course, Microsoft can afford to write these cheques practically ad infinitum, but if your tools are so lacking in perceived utility that you need to bribe people to use them (even if the graft is partially subsidized by affiliate fees), perhaps this is not really the best you could hope for from your marketing team.

You can’t take on Google by trying to buy, or even out-feature, your way into the blank-text-box Search Engine arena. Except for some regional players, like Russia’s Yandex, they’ve won and will not soon be replaced.

What Ballmer, and lots of other people, are missing is that the Search marketplace as we know it is poised for a change. Much of this change emerges from the fact that Google fundamentally owns the global Search Market, but much of the opportunity extant in this space comes from the fact that the technology behind search, and how people will make use of search engines in the future, will be a whole lot different than what you see when you type in www.google.com today.

…. but, there is light at the end of the tunnel for folks who are on the outside looking in at Google’s substantial (and impossible to dislodge) market share:

For most people, web search is a kludge.

Think about how you use Google today. Think about why you type things into that blank text space beckoning to you on your Firefox browser, or why you surf over to Google.com and enter a few snippets of text into that empty area amidst the sea of clutter-free Google whiteness ten, twenty, or maybe many more times per day.

In some cases, you overheard something being discussed in a coffee shop. Or you saw a billboard ad. Something offline motivated you to head to the blank text box and ask it to do your bidding. That is Google’s fundamental market opportunity and has remained largely unchanged since the first search engines began emerging in 1995.

This is, however, just a fraction of the reasons why many of us head to search engines. Often the reasons are as much motivated by inadequate information at one site as by anything else. An example: You’re reading an article from a wire service like Reuters, which rarely include photos, about a car or a submarine or a mountain. You’d like to see what that looks like, so off to Google you go. Or you’re looking at a new LCD on eBay, but the seller hasn’t listed the number and type of inputs that come with it; so off to Google you go to try and find the specifications.

In short, most often we go to Google to search for things because our browsers aren’t good at building pathways between like objects on the web. These types of Searches are what I call context-driven. You shouldn’t need to do this. You shouldn’t need to interrupt your surfing to drop off to a third-party site in order to add flavour to the web objects which have already garnered your interest.

What if you could press a button and instantly be delivered relevant information that is contextual to that which you are/were looking at? What if sites displaying articles from wire services (notable for their sparseness) were able to draw in information – in realtime – which added relevant photos, videos, or related stories?

Some of this is already happening, albeit rather jerkily. One of the leaders which started doing this some time ago was Sphere, which was recently acquired by AOL. It took them some time to draw the same conclusions as I have, and they had a difficult time monetizing these services. But on a great enough scale the same technologies which make relevant content possible also make relevant advertising possible. And while click-thrus will be fewer in quantity they can be greater in quality and therefore infinitely more valuable, thanks to much more accurate targeting.

Being accurate in driving these sorts of searches is hard. Whereas Google relies on its users to sift through its top 30 or so recommendations to find the most relevant information, contextual search engines need to be able to do that with high accuracy on the first few matches with little to no meatware — sorry, Mahalo. Many of the current buzzwordy trends such as the Semantic Web initiatives, Social Search, the shift from RSS to Atom, and API-accessible semantic processing are key enablers to make this easier, but there’s still a considerable amount of R&D necessary to beat Google’s current level of accuracy in this regard.

As a result, you need a long lead to get there, and few of the companies dabbling in the Vertical Search space have raised enough capital or have investors who have committed to developing these opportunities. But in the long run, this will augment Web Search and replace much of the traffic that is today driven by Google’s simple, primitive, empty text box.

What’s clear is that Microsoft’s desperate attempts to lure users to its essentially equivalent service to Google’s can only cost its shareholders. A new paradigm is necessary and, fortunately, the opportunity is ripe for the picking, right in front of us all.

This is a rare opportunity where the solution lies in good, solid R&D and product realization — not in leveraging semi-monopolistic product integration or in brute force advertising spending. Is Microsoft bold enough to understand, and embrace, the fact that Search is shifting? Do they have the product and engineering people to make this happen?