Of course, Microsoft can afford to write these cheques practically ad infinitum, but if your tools are so lacking in perceived utility that you need to bribe people to use them (even if the graft is partially subsidized by affiliate fees), perhaps this is not really the best you could hope for from your marketing team.

You can’t take on Google by trying to buy, or even out-feature, your way into the blank-text-box Search Engine arena. Except for some regional players, like Russia’s Yandex, they’ve won and will not soon be replaced.

What Ballmer, and lots of other people, are missing is that the Search marketplace as we know it is poised for a change. Much of this change emerges from the fact that Google fundamentally owns the global Search Market, but much of the opportunity extant in this space comes from the fact that the technology behind search, and how people will make use of search engines in the future, will be a whole lot different than what you see when you type in www.google.com today.

…. but, there is light at the end of the tunnel for folks who are on the outside looking in at Google’s substantial (and impossible to dislodge) market share:

For most people, web search is a kludge.

Think about how you use Google today. Think about why you type things into that blank text space beckoning to you on your Firefox browser, or why you surf over to Google.com and enter a few snippets of text into that empty area amidst the sea of clutter-free Google whiteness ten, twenty, or maybe many more times per day.

In some cases, you overheard something being discussed in a coffee shop. Or you saw a billboard ad. Something offline motivated you to head to the blank text box and ask it to do your bidding. That is Google’s fundamental market opportunity and has remained largely unchanged since the first search engines began emerging in 1995.

This is, however, just a fraction of the reasons why many of us head to search engines. Often the reasons are as much motivated by inadequate information at one site as by anything else. An example: You’re reading an article from a wire service like Reuters, which rarely include photos, about a car or a submarine or a mountain. You’d like to see what that looks like, so off to Google you go. Or you’re looking at a new LCD on eBay, but the seller hasn’t listed the number and type of inputs that come with it; so off to Google you go to try and find the specifications.

In short, most often we go to Google to search for things because our browsers aren’t good at building pathways between like objects on the web. These types of Searches are what I call context-driven. You shouldn’t need to do this. You shouldn’t need to interrupt your surfing to drop off to a third-party site in order to add flavour to the web objects which have already garnered your interest.

What if you could press a button and instantly be delivered relevant information that is contextual to that which you are/were looking at? What if sites displaying articles from wire services (notable for their sparseness) were able to draw in information – in realtime – which added relevant photos, videos, or related stories?

Some of this is already happening, albeit rather jerkily. One of the leaders which started doing this some time ago was Sphere, which was recently acquired by AOL. It took them some time to draw the same conclusions as I have, and they had a difficult time monetizing these services. But on a great enough scale the same technologies which make relevant content possible also make relevant advertising possible. And while click-thrus will be fewer in quantity they can be greater in quality and therefore infinitely more valuable, thanks to much more accurate targeting.

Being accurate in driving these sorts of searches is hard. Whereas Google relies on its users to sift through its top 30 or so recommendations to find the most relevant information, contextual search engines need to be able to do that with high accuracy on the first few matches with little to no meatware — sorry, Mahalo. Many of the current buzzwordy trends such as the Semantic Web initiatives, Social Search, the shift from RSS to Atom, and API-accessible semantic processing are key enablers to make this easier, but there’s still a considerable amount of R&D necessary to beat Google’s current level of accuracy in this regard.

As a result, you need a long lead to get there, and few of the companies dabbling in the Vertical Search space have raised enough capital or have investors who have committed to developing these opportunities. But in the long run, this will augment Web Search and replace much of the traffic that is today driven by Google’s simple, primitive, empty text box.

What’s clear is that Microsoft’s desperate attempts to lure users to its essentially equivalent service to Google’s can only cost its shareholders. A new paradigm is necessary and, fortunately, the opportunity is ripe for the picking, right in front of us all.

This is a rare opportunity where the solution lies in good, solid R&D and product realization — not in leveraging semi-monopolistic product integration or in brute force advertising spending. Is Microsoft bold enough to understand, and embrace, the fact that Search is shifting? Do they have the product and engineering people to make this happen?

These days, solving the search problem is just one step on a long path to building valuable services that people enjoy and make use of every day. Deep nerds like tackling these issues because they have all the hallmarks of geek chic. They are difficult algorithmically, require massive planning from a scaling perspective, and require constant tweaking. Google was successful at attracting people to its search engine for two reasons: it had a cleaner interface (they hadn’t decided to become a Portal) and it had more accurate results (the other engines had become gamed). I’m sure Google has tons of patents around their search capability however I am too lazy to search for them because sifting through the results separating wheat from chaff would take way too much time.

And that, dear reader, is precisely the point. Google, too, has been gamed — as will every search engine that comes into common use. So what am I on about? Well, Jimmy Wales wants to open-source the search engine… and for the record I think it’s a great idea, and one that threatens GoOgle substantially.

My logic is this: If the value of a search engine is no longer the search engine itself, but instead the application to which it is applied, then why not accept its value as a generic must-have and open-source the thing? We can all benefit from the assumption that the search engine itself will always be gamed by spammers and sploggers and search engine marketers. Once we do that, creating a community that is invested in the efficacy of the search engine (because they’re making money from it) also creates a system by which that community is incentivized to keep the thing working properly as it’s gamed by persistent SEO gremlins. This is far more effectively done by a collective of companies than it is by a bunch of companies tweaking their own engines independently, pursuing near-term, interim, proximate advantages.

Wikia Search is nudging closer to existence, but I think it’s applying the brute-force labour at the wrong end. As Jimmy Wales becomes more and more assertive and aggressive with his crusade to fix search using an army of lemmings using the Wikipedia recipe, which means he’ll use the community at large to determine the merit of matches found by his search engine, he’s extending the Wikipedia model to searching. Users will “vote” matches to the top of the rankings.

interesting notion, but I think he has the right idea but might be missing the mark on execution. As anyone who’s watched Sanjaya on American Idol can attest, user-voting is not always the most expeditious method of ensuring quality. Wikipedia uses a broken-source (have I just coined this term?) publishing model: it achieves one thing very well (aggregating information and content from diverse sources) at the expense of the other (ensuring that information is trustworthy, balanced or factually correct is problematic). Applying this model for Search is therefore solving the easy problem (search engines already aggregate and index things quite well) with the wrong method (envision Wikia Search gaming teams in Bangladesh sweat shops “voting up” rankings for their customers on the engine).

So, right idea — wrong solution. Let’s create an engine that everyone can (and does) use, that everyone can tweak and repair, and that is policed by a foundation which has as its only goal the efficacy of the product. The Deep Thinker Nerds who like to fiddle with these kinds of problems will be attracted naturally to the project, and their incomes could easily be supported by the companies benefiting from the expansion of the technology. Jimmy’s in a position to lead this, to some degree, but he doesn’t evidently understand that the strength of Wikipedia will be the achilles heel of this project. He has claimed the high ground but I fear that he will inevitably fail.

-Ian.

-Ian.

—– http://story.news.yahoo.com/news?tmpl=story&cid04&ncids8&e=6&u=/ washpost/20030724/tc_washpost/a32817_2003jul23 AOL Subscribers Down by 846,000 Thu Jul 24,12:23 AM ET

Add Technology – washingtonpost.com to My Yahoo!

By David A. Vise, Washington Post Staff Writer

America Online’s subscriber base plunged by 846,000 in the second quarter, as hundreds of thousands left for cheaper or faster Internet connections and a similar number were dropped because they had been mistakenly counted in the past, AOL Time Warner Inc. disclosed yesterday.

In addition, new disclosures about a federal investigation into improper accounting at Northern Virginia-based America Online Inc. showed that the division’s legal problems are hurting other parts of the AOL Time Warner media empire.

AOL Time Warner said yesterday that the Securities and Exchange Commission ( news -web sites ) would not allow it to spin off a portion of its cable television unit until it resolves a dispute over how to account for hundreds of millions of dollars in questionable revenue from a complex deal with German media firm Bertelsmann AG ( news -web sites ).

AOL Time Warner also said it may restate previously reported profits and sales linked to the Bertelsmann transaction. And the company indicated that it could not determine how long the SEC and Justice Department ( news -web sites ) investigations into its bookkeeping practices will last.

The company said its profit increased to $1.1 billion (23 cents per share) in the second quarter, from $396 million (9 cents) in the second quarter of 2002. Revenue increased about 6 percent, to $10.8 billion. The profit figure included a number of substantial one-time gains from the settlement of a lawsuit with Microsoft Corp. and the sale of various businesses.

Despite solid results in divisions other than America Online, AOL Time Warner shares fell yesterday by $1.14, or 6.8 percent, to $15.71, as analysts and major investors reacted to the continuing uncertainty caused by the SEC investigation, the threat of increasingly costly shareholder lawsuits, the deterioration in America Online’s performance, and disappointment that the strength of AOL Time Warner’s film, publishing and cable television operations did not prompt the company to substantially increase its financial projections.

“Our goal for the remainder of this year is to keep laying the foundation that will enable us to exit 2003 with more momentum than we had when we entered it, with an eye toward achieving, strong sustainable growth next year and beyond,” said Richard D. Parsons, chairman and chief executive of AOL Time Warner.

AOL, the nation’s biggest Internet service provider, has shed a total of 1.2 million subscribers over the past year and now has 25.3 million subscribers in the United States.

The company said the total includes 2.2 million high-speed subscribers, an increase of 300,000 over the past three months. During that period, AOL launched an enhanced high-speed offering and promoted it with an advertising campaign titled, “AOL for Broadband: Welcome to the World Wide Wow.”

In addition to losing dial-up subscribers faster than expected, AOL is predicting that its online advertising revenue will drop about 40 percent in 2003. The decline is occurring even though the total dollars spent on advertising online is growing nationally, a trend that can be seen in the financial results of some of America Online’s competitors, including search engines Yahoo and Google and many specialized Web sites.

AOL Time Warner had sought to persuade SEC investigators that they were mistakenly challenging the accounting for the two-part Bertelsmann deal. But the company said yesterday that the commission has refused to back down.

“The company and its auditors continue to believe the accounting for those transactions is appropriate, but it is possible that the company may learn additional information as a result of its own review, discussions with the SEC and/or the SEC’s ongoing investigation that would lead [AOL Time Warner] to reconsider its views,” the firm disclosed.

The Bertelsmann deal involved AOL’s sale of roughly $400 million in advertising to Bertelsmann in connection with the purchase of Bertelsmann’s stake in AOL Europe.

AOL Time Warner released its second-quarter results prior to the opening of stock trading yesterday morning. Although it cut its projections for America Online, the company beat Wall Street estimates as its cable television, motion picture and publishing businesses thrived.

“Our solid results in this quarter and the first half of the year give us confidence that we can deliver on all of our 2003 financial objectives,” Parsons said. He added that the company is continuing to reduce its hefty debt through the sale of businesses and the spending of billions of dollars of excess cash generated by operations.

The Warner Brothers and New Line Cinema movie units generated $572 million and $239 million, respectively, at the box office in the United States. “The Matrix Reloaded” led the way among new releases, while “Harry Potter ( news -web sites ) and the Chamber of Secrets” boosted DVD and CD sales.

“On balance,” said Deutsche Bank, “we think this report is good news.”

In a conference call with analysts, Parsons said he was no longer counting on the sale of stock in Time Warner Cable to generate cash for debt reduction this year. Instead, he said, the handling of any cable spinoff will be determined by broader issues, including the best way to help that subsidiary grow.

“The specific timetable for executing an IPO will depend on strategic considerations, not balance sheet imperatives, as well as the status of our SEC investigations,” Parsons said.

Google vs. Evil

The world’s biggest, best-loved search engine owes its success to supreme technology and a simple rule: Don’t be evil. Now the geek icon is finding that moral compromise is just the cost of doing big business.

By Josh McHugh

Life used to be so much easier for Sergey Brin. In the autumn of 1998, he and Larry Page unleashed Google with a clear mission: Help computer users find exactly what they want on the Internet. Newbies flocked to the site, grateful for a simple search engine that was both powerful and intuitive. More sophisticated techies came to appreciate Google’s computational elegance and its willingness to shun the “portal” model that crammed ecommerce down their throats. Within months, Google became one of the most popular sites on the Web – and not long after that, “Google” became a verb. Today, Internet users spend about 15 million hours a month on the site. Google.com logs more than 28 million visitors each month, nearly as many as Yahoo! and MSN. Nearly four out of five Internet searches happen on Google or on sites that license its technology.

Google owes its swelling popularity to deft algorithms that quickly divine what’s useful on the Web. But there’s more to it than that. At Google, purity matters. Over the years, Brin and Page have resisted pressure to run banners, opting instead for haiku-like text ads and unintrusive sponsored links. They’ve taken a stand against pop-ups and pop-unders and refused ads from sites they consider to be overly negative. All the while, they’ve stubbornly kept the Google homepage concise and pristine. On just a faint whisper of a marketing campaign, the company pulled in an estimated $70 million last year (a third from licensing fees and the rest from ads).

The Google strategy appeals to every engineer’s sense of The Way It Should Be. Build the best entry in the science fair. Do not tart it up. Do not make it more clever than it needs to be.

But a funny thing is happening on the way to Internet adulthood – Google’s awkward teen years. The company’s growth spurt has spawned a host of daunting questions that no data-retrieval system can easily answer. Should Google play ball with repressive foreign governments? Refuse to link users to “hate” sites? Punish marketers who artificially inflate site rankings? Fight the Church of Scientology’s attempts to silence critics? And what to do about the cache, Google’s archive of previously indexed pages? In April, the German national railroad threatened legal action to remove an obsolete site containing sabotage instructions.

Most major companies refer to a detailed code of corporate conduct when considering such policy decisions. General Electric devotes 15 pages on its Web site to an integrity policy. Nortel’s site has 34 pages of guidelines. Google’s code of conduct can be boiled down to a mere three words: Don’t be evil.

Very Star Wars. But what does it mean?

“Evil,” says Google CEO Eric Schmidt, “is what Sergey says is evil.”

Of the Google triumvirate, Schmidt makes sure the company stays on course financially and strategically; Page keeps busy in the R&D lab, cranking out new features; and the 29-year-old Brin, in his role as Google’s conscience and head policymaker, spends his days gripping the moral tiller – and in so doing, imposes his worldview on everyone else.

That puts Brin at the flashpoint of most of the major Internet-related controversies. He knows his decisions have far-reaching consequences. He feels the pressure that attends Google’s growing power. “I do get fairly stressed,” Brin says. “I’d like to feel a little less scrutinized.”

Google has succeeded by adhering to one, pure principle: Do good by users. Now, for the first time in its history, Google is facing rifts between what’s good for users and what’s good for Google. And Sergey Brin is finding that purity just doesn’t scale.

II.

Don’t be evil. Brin has had to refer back to those three words quite a bit over the past year. Governments, religious bodies, businesses, and individuals are all bearing down on the company, forcing Brin to make decisions that have an effect on the entire Internet. “Things that would normally be side issues for another company carry the weight of responsibility for us,” Brin says.

In March, lawyers representing the Church of Scientology requested that Google stop linking to a Norwegian anti-Scientology site called Operation Clambake. The church claimed the site, xenu.net, displayed copyrighted Scientology content and that by providing links to the information, Google was in violation of the Digital Millennium Copyright Act. Much to the dismay of many First Amendment fans, Google caved, removing the offending pages from its index.

In May, Anita Roddick, the outspoken British founder of the Body Shop, blasted Google in her blog for yanking a text ad for her site. Google’s explanation: Roddick had called actor John Malkovich a “vomitous worm” in her blog, violating a Google policy against accepting ads for sites that are “anti-” anything. After Roddick protested, Google offered to reinstate the ad in exchange for a promise from Roddick that she would remove the Malkovich reference from the first page of her site. When she refused, Brin had a decision to make: Should he give in and accept Roddick’s money, or stand by his principles? He chose his principles.

Three months later, Daniel Brandt, who runs google-watch.org, attacked PageRank, the algorithm at the heart of Google’s vaunted system, accusing the company of being unfair and undemocratic. Brandt urged the FTC to investigate Google and regulate it as a public utility – as a company that, in effect, controls access to the Internet’s natural resources. The mainstream press tended to dismiss Brandt as a webmaster spurned by a low Google ranking, but in the online forums and weblogs, many agreed with his assertion. As far as search engines go, Google has become the only game in town.

Then in the first week of September, Brin found himself pulled into matters of foreign policy. He received several emails from users telling him that the Chinese government, worried about political dissent in the weeks before the 16th Chinese Party Congress, had shut down access to the site. “Our Chinese traffic was down by a factor of five,” Brin says. “We were blocked.”

Brin was no expert on international diplomacy. So he ordered a half-dozen books about Chinese history, business, and politics on Amazon.com and splurged on overnight shipping. He consulted with Schmidt, Page, and David Drummond, Google’s general counsel and head of business development, then put in a call to tech industry doyenne Esther Dyson for advice and contacts. Google has no offices in China, so Brin enlisted go-betweens to get the message to Chinese authorities that Google would be very interested in working out a compromise to restore access. “We didn’t want to do anything rash,” Brin says. “The situation over there is more complex than I had imagined.”

Four days later, Chinese authorities restored access to the site. How did that happen? For starters, the Chinese government was deluged with outcries from the nation’s 46 million Internet users when access to Google was cut off. “Internet users in China are an apolitical crowd,” says Xiao Qiang, executive director of New York-based Human Rights In China. “They tend to be people who are doing well, and they don’t usually voice strong views. But this stepped into their digital freedom.”

The quick workaround: Chinese authorities tweaked the national firewall, making the new Google China different from the site that was turned off. Today, Chinese who use Google to search on terms like “falun gong” or “human rights in china” receive a standard-looking results page. But when they click on any of the results, either their browsers are redirected to a blank or government-approved page, or their computers are blocked from accessing Google for an hour or two. “They have a new mechanism that can block the results of certain searches,” Brin says. Did Google help China find or obtain the filtering technology? “We didn’t make changes to our servers” is all he’ll say.

In late October, a report by two Harvard researchers revealed that Google had begun filtering its own servers to block users in Germany, France, and Switzerland from accessing sites carrying material likely to be judged racist or inflammatory in each country. Neither Brin nor anyone else at Google will talk about about the preemptive self-censoring moves in Europe.

In the wake of these international incidents, members of Google’s loyal, tech-savvy constituency began to question the company’s motives. “I am a little on the fence about Google’s latest actions,” wrote Brian Osborne, a staff writer for Geek.com, a news site. “On one hand, I understand Google’s stance that it must remain in compliance with German and French laws. Nevertheless, Google is putting itself on a very slippery slope.”

III.

“What is this?” asks a visitor squinting at the form he must sign before proceeding to the cafeteria at Google’s Mountain View, California, headquarters. “An NDA? To have lunch?”

The receptionist shrugs. “This is Google,” she says. “They’re crazy that way.”

The Googleplex, contrary to almost every written account of the place, is hardly a haven of easygoing geek whimsy. The cafeteria is adorned with a tie-dyed banner, but the Google employees aren’t humming any Dead songs. Most of them appear deadly serious. Brin’s second-floor office overlooks a courtyard festooned with empty hammocks. A book about Enron rests on his coffee table.

Brin’s designation as Google’s policy maven is relatively new. He, the big thinker, and Page, the mad scientist, complemented each other and shared nearly every role in Google’s early years. “Larry was always the driver,” says Scott Hassan, who did much of the programming for the original Google. “A big part of his role was going around and yelling ‘Why can’t it do this? Why isn’t this working?'” Brin would sit next to Hassan and watch him write code, pointing out errors and taking an occasional turn at the keyboard.

The frenetic Page looked at all the popular engines at the time and decided they were going about search the wrong way. By relying on HTML code – meta tags as well as page text – they would bring back all sorts of irrelevant information and open themselves up to massive manipulation by webmasters looking to increase their own rankings. Brin took Page’s observation and ran with it. He figured the best way around the problem was to harness the vast repository of human judgments already preserved on the Internet in the form of hyperlinks. “Most people search for local maximums – like figuring out how to get the best car, the best immediate situation,” Hassan says. “Sergey is always searching for global maximums.”

By 2001, Google’s breakneck growth convinced Page and Brin it was time to establish a more rigid structure. Page handed over the CEO title to Schmidt and became copresident with Brin. The move freed up Page to focus on developing his knack for product development (as a child, he crafted a printer out of spare parts and Lego blocks). Brin’s passion for the big picture made him the natural choice to spend time on Google’s growing role in the world.

Which means Brin’s views on politics and policy matter quite a bit. Not that he’s willing to talk. He tells me he listens to NPR on his morning drive to work. I think Democrat and ask about his voter affiliation. He says he votes across party lines. Independent? He smiles and tells me there’s no easy shortcut toward figuring out how he comes to his decisions about good and evil. And even if there was, he wouldn’t let me in on it. If I succeed in figuring out exactly what he considers good and evil, people who don’t care about Google users might start gaming him the way they try to game his search engine.

Born in Moscow and raised in the suburbs of Washington, DC, Brin grew up listening in on conversations at the dinner parties thrown by his father, a math professor, and his mother, a NASA scientist. Talking about his decisions and the values he holds most dear, Brin chooses his language carefully, but one word he repeatedly comes back to is “useful.” And while Google’s policy decisions over the past year look a bit haphazard at first glance, they begin to make more sense in a worldview where usefulness is the paramount virtue.

Aside from the indisputable goodness of causing hard-line Communist Party officials to say the word “Google” to one another for a few days, it’s difficult to say on which side of the good-evil line the company’s China resolution falls. Brin seems at peace with how it all turned out. “Political searches are not that big a fraction of the searches coming out of China,” he says. “You want to look at the total value picture that a search engine like Google brings and think of all that it’s used for.”

But Xiao Qiang, the activist, thinks the company should have taken a firmer stand. “Ultimately, China’s state censorship mechanism will have to submit to this growing demand for freedom from Chinese netizens,” Xiao says. “It’s important to protect integrity, particularly for an Internet firm.”

On the same day that China blocked access to Google, it also flipped the switch on AltaVista. AltaVista issued a defiant statement to the media and went on to list several ways to access the site. Months later, AltaVista is still blocked. Brin figures that by meeting China halfway, Google remained available – and useful – to visitors and also preserved its advertising revenue there. “You have to look at the total value picture,” he says.

What about the Scientology mess? Didn’t Google give in too easily? Jennifer Urban, a fellow at Berkeley’s Boalt Hall School of Law and a member of Chilling Effects, an organization formed to document attempts to stifle speech on the Internet, says that from a legal standpoint, Google’s hands were tied. “To qualify for safe harbor protection from liability, they really have to err on the side of taking down the link,” Urban says.

In fact, Google didn’t fold entirely. After consulting with Brin, Kulpreet Rana, Google’s head of IP, found a way that Google could comply with the law without letting the Scientologists erase their critics from the Internet. The solution: When Google gets a request to remove a link under the safe harbor provisions of the DMCA Section 512, it substitutes a link to a form on the Chilling Effects’ site. The form contains the Web address of the page in question, and anyone still interested in the site can direct their browser to the address.

Does abiding by the letter of a bad and flimsy law absolve Google from charges that it squashed free expression? Cindy Cohn, legal director of the Electronic Frontier Foundation, is certain that a vigorous legal challenge would put an end to the steady flow of Section 512 filings Google receives but admits she doesn’t expect Google to devote its resources to such a broad fight. And while some cheered Google’s workaround as evidence of a rebellious bit of payback – a small point scored against the enemies of unfettered speech – the move is another instance of Brin choosing the path of usefulness over a righteous crusade.

IV.

If Brin’s code of good and evil permits the company to negotiate with sovereign governments and allows for some legal meddling from unpopular religions, there is no wiggle room – no gray area whatsoever – when it comes to those who attempt to subvert the power of Google to their own commercial ends. One thing Brin is sure of: On the side of evil lies trickery.

I ask Brin to imagine, for a moment, running his company’s evil twin, a sort of anti-Google. “We would be doing things like having advertising that wasn’t marked as being paid for. Stuff that violates the trust of the users,” he says, describing a site that sounds not unlike the pay-for-placement search site Overture. “Say someone came looking for breast cancer information and didn’t know that some listings were paid for with money from drug companies. We’d be endangering people’s health.”

The anti-Google might also be more amenable to the growing business of “optimization,” the altering of Web sites so that they rank higher in search engine results. For a fee, there’s help for a Dallas plumber who’s unhappy that his site is on the 17th page of results when someone types “Dallas plumber” into Google. An optimizer will tweak the site in such a way that boosts it to, say, the 3rd page of results.

To pull this off with Google, an optimizer needs to understand how the company’s search mechanism works. Google uses 100 or so closely guarded algorithms to determine its search results. The best known of the lot is called PageRank, which allocates relevancy to a page according to the number and importance of pages linked to it, the number and importance of pages linked to each of those pages, and so on. One ploy is to create “link farms,” in which an optimizer gets clients to link to one another, racking up relevancy points. In general, optimizers make a living by guessing what Google regards as important. The way Brin sees it, the optimizers are co-opting Google’s bond of trust with its users. He regards optimizers the way a mother grizzly might regard a hunter jabbing at her cub with a stick.

Every month, when Google updates its index and its mix of algorithms, it rakes a disruptive claw across the optimizers’ systems. In the industry, the monthly shuffle is known as the Google Dance, and Brin doesn’t mind letting on that if Google ends up dancing all over the optimizers, so much the better. “When we change and improve our technology, things get shuffled around,” Brin says, “and sometimes it has a disproportionate effect on optimization sites.”

Consider the case of Bob Massa, a former solid oak dining room furniture salesman who lives in Oklahoma City and runs SearchKing, an optimization company he started in 1997. Last summer, Massa received a rare gift from Google in the form of the Google Toolbar, a software program that lets users perform searches without going to Google.com. More important for Massa, the Toolbar shows the approximate PageRank, on a scale of one to ten, of whatever page a user is visiting. It was the first time since Brin and Page were in grad school that they’d shared so much technical information. After years of watching Google’s every move like an Etruscan high priest trying to augur divine intent from cloud formations, Massa had a piece of the goods. On August 9, Massa started selling optimization based on PageRank.

After the Google Dance of September 20, most of Massa’s customers suddenly found themselves in a heap at the very bottom of Google’s 3 billion site index. It seems that the improvements Google had made included a severe downgrade of sites with links to SearchKing. Massa’s customers, needless to say, were very, very unhappy. “Everyone thinks I’m the biggest idiot in the world for making Google mad,” Massa said in October.

He filed suit a few weeks later, charging that Google downgraded his customers’ scores in a deliberate attempt to put him out of business. The suit asks for an injunction forcing Google to restore the scores to pre-Dance levels, and seeks $75,000 in damages. “It’s a classic good versus evil thing,” says Massa, turning Brin’s framework back on Google itself. “I knew they wouldn’t like it. I didn’t think they’d go so far as to wipe out all these little people.”

The day Massa’s suit was filed, the reaction from the Slashdot crowd and most other forums was predictably vociferous, with posters stumbling over themselves to craft metaphors painting Massa as a criminal suing his victim. But gradually, a surprising number of people, while careful not to look as though they were defending Massa, began tagging the search engine as a Google-opoly. It’s hard to sympathize with a David as parasitic as Massa, but Slashdotters tend to be uneasy with Goliaths of any stripe, especially when their methods are kept secret.

And the real problem with Massa is that he’s simply the termite Brin is able to see. There are thousands more behind the wall, invisibly boring away at the very structure of Google’s house. “It’s easy to become overly obsessed with those kinds of things,” Brin admits.

It would make things a lot easier for Brin if the world’s webmasters would just act as though his site didn’t matter, but that’s not human nature. There’s no way around it – as long as Google remains the search engine of choice, the arms race between Google coders and the hordes of optimizers will go on.

V.

As proficient as Google is at revealing information, Brin is adept at keeping key morsels under wraps. In a way, that makes a lot of sense. Although the obvious image of Google is one of accumulation, the essence of data retrieval is just the opposite. Google is about division and subtraction, narrowing down billions of choices before revealing the most promising. Brin’s world isn’t as simple as visible equals good, hidden equals evil. Google’s effectiveness as a search tool depends largely on how well it’s able to shroud the site’s inner workings from the commercial interests that clutter so much of the Internet today.

But here’s the thing: If Brin thinks his job has become more difficult over the past year, it may soon become near impossible. In September, at the height of the China controversy, Google legal eagle Drummond spotted an article about the prospect of a Google IPO, which, the story said, might be the spark to ignite the dormant public offerings market. Drummond forwarded the story with some sardonic comments. In his office, Brin tries to find the email for me but can’t. He notes the irony in that, and goes on to paraphrase the note: “Oh, OK, now we’re going to reform the Chinese government – and on top of that, we’re going to fix Wall Street.”

Schmidt claims the company is in no rush to go public, but his appointment and the hiring of CFO George Rayes last August were unmistakable steps in that direction. When the IPO comes, it will bring riches – and more problems.

As a private company, Google has one master: users. As a public company, there are shareholders to worry about. And more than happy users, shareholders want ever-greater profits. Thus far, Brin and Page have succeeded in standing up to pressures that might compromise Google and the user experience. Google’s influential stand against pop-up ads extends beyond its own domain – the company rejects advertisers whose links take Google users to pages that feature pop-ups. (AOL followed suit in October, announcing its own pop-up moratorium.) But when Google becomes a public company, shareholders might force the site to take a more amenable position, if the price is right. After all, for several years, Yahoo! refused to accept anything but fast-loading banner ads, claiming that it was looking out for users. That policy lasted until right about the time that the company’s stock price began to cave.

Such pressure could cause Brin to rethink other policies, like his decision to refuse all alcohol and tobacco advertising. The fact that Google accepts advertising for adult content sites is an intriguing commentary on Brin’s morality: Cigarettes and booze are evil; porn is not. It’s a policy that would become progressively harder to defend were Google to go public. Then there’s the Google cache to consider. Today’s users love having access to a warehouse of information that was once published on the Internet but has since disappeared. Some information goes away for a reason, though. The cache could get Google in trouble, and Brin & Co. could soon find themselves facing all sorts of libel, defamation, or copyright lawsuits.

Increased competition may also cause Brin to do other things he’s loath to do. So far, Google has gotten by without much in the way of competition from the other Internet superpowers. But in May, Yusuf Mehdi, the head of MSN, said he views Google as “more of a competitor than a partner” in the effort to become the default homepage on millions of browsers. What if, as Google.com solidifies its position as the focal point of the Internet, Yahoo! and AOL begin to rethink the millions in licensing fees they pay to what has become a top competitor? Brin may be forced to make the kind of concessions that he’s thus far reserved for international governments.

The utilitarian manner in which Google has achieved its success has made it a sentimental favorite among the code-parsing set. Tech-community sites like Slashdot are almost uniformly pro-Google. Those with the temerity to bring lawsuits against Google ultimately feel the burn of online flames, watching their servers wither under the quasi-zealous wrath of thousands of engineers defending one of their own. But as Google is forced to make more concessions to realpolitik, its bonds with that idealistic constituency will inevitably continue to fray.

And without any sort of technological lock-in, it would be very easy for Google’s visitors to simply start using other search engines. Fast Search & Transfer, based in Norway, boasts a 2.1 billion-page index at www.alltheweb.com, and its search engine works as quickly as Google’s. What’s more, it does a complete crawl of the Internet every 7 to 11 days compared with Google’s 28 days. What if an influential group of politically active netizens makes a rousing case for boycotting Google on the grounds that it is anti-free speech and in cahoots with repressive governments? How long can a hugely powerful company that plays its decisions so close to the vest and refuses to justify itself publicly count on the devotion of the average information-hungry Web user?

It’s inevitable that a company of Google’s size and influence will have to compromise on purity. There’s a chance that, in five years, Google will end up looking like a slightly cleaner version of what Yahoo! has become. There’s also a chance that the site will be able to make a convincing case to investors that long-term user satisfaction trumps short-term profit. The leadership of the Internet is Sergey Brin’s to lose. For now, at least, in Google we trust.

Josh McHugh (josh [at] buzzkiller [dot] net) wrote about Wi-Fi campus life in Wired 10.10.

———–

http://www.google.com/technology/pigeonrank.html

The technology behind Google’s great results

As a Google user, you’re familiar with the speed and accuracy of a Google search. How exactly does Google manage to find the right results for every query as quickly as it does? The heart of Google’s search technology is PigeonRank, a system for ranking web pages developed by Google founders Larry Page and Sergey Brin at Stanford University.

Building upon the breakthrough work of B. F. Skinner, Page and Brin reasoned that low cost pigeon clusters (PCs) could be used to compute the relative value of web pages faster than human editors or machine-based algorithms. And while Google has dozens of engineers working to improve every aspect of our service on a daily basis, PigeonRank continues to provide the basis for all of our web search tools.

Why Google’s patented PigeonRank works so well

PigeonRank’s success relies primarily on the superior trainability of the domestic pigeon (Columba livia) and its unique capacity to recognize objects regardless of spatial orientation. The common gray pigeon can easily distinguish among items displaying only the minutest differences, an ability that enables it to select relevant web sites from among thousands of similar pages.

By collecting flocks of pigeons in dense clusters, Google is able to process search queries at speeds superior to traditional search engines, which typically rely on birds of prey, brooding hens or slow-moving waterfowl to do their relevance rankings.

When a search query is submitted to Google, it is routed to a data coop where monitors flash result pages at blazing speeds. When a relevant result is observed by one of the pigeons in the cluster, it strikes a rubber-coated steel bar with its beak, which assigns the page a PigeonRank value of one. For each peck, the PigeonRank increases. Those pages receiving the most pecks, are returned at the top of the user’s results page with the other results displayed in pecking order.

Integrity

Google’s pigeon-driven methods make tampering with our results extremely difficult. While some unscrupulous websites have tried to boost their ranking by including images on their pages of bread crumbs, bird seed and parrots posing seductively in resplendent plumage, Google’s PigeonRank technology cannot be deceived by these techniques. A Google search is an easy, honest and objective way to find high-quality websites with information relevant to your search.

Data

PigeonRank Frequently Asked Questions

How was PigeonRank developed?

The ease of training pigeons was documented early in the annals of science and fully explored by noted psychologist B.F. Skinner, who demonstrated that with only minor incentives, pigeons could be trained to execute complex tasks such as playing ping pong, piloting bombs or revising the Abatements, Credits and Refunds section of the national tax code.

Brin and Page were the first to recognize that this adaptability could be harnessed through massively parallel pecking to solve complex problems, such as ordering large datasets or ordering pizza for large groups of engineers. Page and Brin experimented with numerous avian motivators before settling on a combination of linseed and flax (lin/ax) that not only offered superior performance, but could be gathered at no cost from nearby open space preserves. This open space lin/ax powers Google’s operations to this day, and a visit to the data coop reveals pigeons happily pecking away at lin/ax kernels and seeds.

What are the challenges of operating so many pigeon clusters (PCs)?

Pigeons naturally operate in dense populations, as anyone holding a pack of peanuts in an urban plaza is aware. This compactability enables Google to pack enormous numbers of processors into small spaces, with rack after rack stacked up in our data coops. While this is optimal from the standpoint of space conservation and pigeon contentment, it does create issues during molting season, when large fans must be brought in to blow feathers out of the data coop. Removal of other pigeon byproducts was a greater challenge, until Page and Brin developed groundbreaking technology for converting poop to pixels, the tiny dots that make up a monitor’s display. The clean white background of Google’s home page is powered by this renewable process.

Aren’t pigeons really stupid? How do they do this?

While no pigeon has actually been confirmed for a seat on the Supreme Court, pigeons are surprisingly adept at making instant judgments when confronted with difficult choices. This makes them suitable for any job requiring accurate and authoritative decision-making under pressure. Among the positions in which pigeons have served capably are replacement air traffic controllers, butterfly ballot counters and pro football referees during the “no-instant replay” years.

Where does Google get its pigeons? Some special breeding lab?

Google uses only low-cost, off-the-street pigeons for its clusters. Gathered from city parks and plazas by Google’s pack of more than 50 Phds (Pigeon-harvesting dogs), the pigeons are given a quick orientation on web site relevance and assigned to an appropriate data coop.

Isn’t it cruel to keep pigeons penned up in tiny data coops?

Google exceeds all international standards for the ethical treatment of its pigeon personnel. Not only are they given free range of the coop and its window ledges, special break rooms have been set up for their convenience. These rooms are stocked with an assortment of delectable seeds and grains and feature the finest in European statuary for roosting.

What’s the future of pigeon computing?

Google continues to explore new applications for PigeonRank and affiliated technologies. One of the most promising projects in development involves harnessing millions of pigeons worldwide to work on complex scientific challenges. For the latest developments on Google’s distributed cooing initiative, please consider signing up for our Google Friends newsletter.

The site is a tad slow, but it truly could be the next Napster model that scares the crap out of the movie industry.

-Ian.

—- $1 films spook Hollywood By John Borland Staff Writer, CNET News.com February 7, 2002, 4:00 AM PT http://news.com.com/2100-1023-831383.html

A Taiwanese Web site is offering hundreds of videos on demand for just $1 each, trumping Hollywood’s plans to deliver similar services and raising the specter of a new round of international copyright battles.

Despite claims that their Movie88 site is following all local copyright laws, the owners of this new venture are drawing scrutiny from a skeptical Hollywood. Meanwhile, Web surfers drawn by free or cut-rate movies are flocking to the site, overloading servers and clogging data pipes.

Like Napster before it, the site is more than just an everyday pirate Web site: It’s a commercial video-on-demand service that’s comprehensive and easy to use. And it works. With the studios’ film services still in development, that’s a dangerous combination of features.

“It’s not a good sign,” said Ken Jacobsen, the Motion Picture Association of America’s director of worldwide piracy enforcement. “But we will deal with these sites rapidly.”

Movie88 and a handful of other sites and services popping up around the world offer growing proof that the movie industry cannot count on immunity from the digital forces that burst in on an unprepared record industry barely two years ago. Film studios had been insulated partly by the enormous size of high-quality digital video files, which made it relatively difficult to trade movie files over the Internet. But that barrier has been progressively falling thanks to improvements in digital video formats and streaming technology.

Such sites are also a sign that many of the most critical copyright battles are shifting overseas. Legal fights in the United States have laid down preliminary ground rules, even if those lawsuits aren’t over. That’s not yet the case in regions with different laws and different court systems, where broadband data pipes and programming talent abound.

That makes life considerably harder for copyright holders.

“There’s a law of diminishing returns in terms of pursuing and prosecuting these things overseas,” said Aram Sinnreich, a Jupiter Media Metrix analyst. “It’s a lamentable situation for studios, but I think the best thing they can do is develop their own legal alternative to draw people.”

Crossing borders Copyright battles overseas have been developing slowly but steadily, even as the most high-profile Internet copyright court cases have remained in the United States.

Some of the clearest signs have come from the names on the software used and distributed online: The author of the DeCSS program that allegedly can be used to make copies of DVDs was Norwegian. Many of the successors to that software program are distributed on German sites. Russian-language search engines provide a hub for people looking for copyrighted music, software and movies.

Only recently have overseas Net companies been targets in high-profile movie or record industry lawsuits, however. Kazaa, a Dutch company, and Bermuda-based Grokster were both sued by the entertainment trade associations last year for operating peer-to-peer file-swapping networks similar to Napster. But even in that case, the companies were sued in a Los Angeles court.

The problem is, international enforcement is tricky. Most developed nations are parties to international trade agreements that give copyright holders rights similar to what they have in the United States. But legal process, details of copyright registration or notification procedures, and even diplomatic pressures can make enforcement a thorny task.

Some of the groundwork for the new Internet battles has been laid. Several years ago, the MPAA conducted a legal survey predicting which 30 countries were most likely to be Net piracy problems, Jacobsen said. They then hired legal teams to start studying local copyright laws.

Now the MPAA and the record companies use search engines to track down Web sites and servers offering copyrighted materials; they use the legal channels they’ve developed to ask Internet service providers to take down the pirated material.

Kazaa and Grokster required taking this approach a step farther: into court. Movie88 may be the next test.

The next battleground Taiwan’s Movie88 appears to be the most ambitious commercial video service yet to hit the Net, with a huge catalog of English-language, Chinese and Japanese movies that can be streamed on demand at fairly high quality.

The site–whose owners declined to respond to e-mail questions, citing the advice of their lawyers–has jumped the gun on an idea already being pursued by the big movie studios. Movielink and Movies.com, each backed by a coalition of studio conglomerates, are preparing Internet-based video-on-demand services that would provide members with access to a large catalog of films for a relatively small fee.

Movie88’s service looks much like a video rental store. It offers a huge range of films that have been released to video, although it appears to lack the first-run films that are often available online or on pirated DVDs hawked on street corners in some Asian cities. The site says it will charge $1 per movie but give each customer a $5 credit for signing up. At least in its English version, it doesn’t yet appear to have the capability to accept credit cards and add more credit to an account.

The movies themselves are streamed in RealNetworks’ RealVideo format and cannot easily be saved or downloaded to a computer’s hard drive. Each movie is available for three days; afterward, it must be renewed with another $1 credit to be seen again.

Early use of the company’s service did display some serious glitches. One attempt by CNET News.com to set up an account and watch the Universal Pictures film “12 Monkeys” worked perfectly. The next day, the same account’s username appeared to be linked to a different account with more than $100 in credit.

The site claims it is operating legally under Taiwanese copyright law by allowing viewers access to the movies for just three days at a time.

“If you are a copyright owner of any materials, movies and films used in Movie88.com, and you feel that your copyright is protected in the Republic of China, kindly contact us,” a note on the site reads. But “the submission…is without our admission to any infringement and/or liability whatsoever.”

But copyright holders’ groups say they haven’t given their permission, and that means a fight is likely brewing. A spokesman for the International Intellectual Property Alliance (IIPA), whose membership list includes the MPAA, said Movie88.com was “clearly a pirate site.”

The MPAA itself stops short of that language, but spokeswoman Emily Kutner says the group is “looking into the site so we can take appropriate steps to deal with it.”

Taiwan has suffered considerable criticism from the United States and other Western countries for being a source of pirated CDs and videos, but it has changed its copyright laws several times in recent years to participate in international trade agreements. It now has essentially the same standards as the United States, and studios would gain full copyright protection there as soon as they release a movie in American theaters, copyright experts said.

Enforcement of copyright laws has been somewhat weaker on the street level, however. This issue goes to the heart of the difficulties copyright holders may see as they increasingly move overseas.

“You see an awful lot of problems, really on the level of local judges and prosecutors,” said Laura Young, a partner at San Francisco law firm Wang & Wang, who has extensive experience in Taiwan. Often “they don’t want to prosecute a local (citizen) for violating the rights of a multinational corporation.”