By Andrew Goodman, 9/29/2002

While researching this feature on Infospace’s recent contributions to the metasearch space, I decided to reacquaint myself with Metacrawler's advanced features. Not a bad idea considering how things have changed. The current media obsession with corporate wrong-doing has found its way into the search field. If you believe everything you read, you’d probably think that there is absolutely no way to use a cool Internet search tool without being bowled over with paid advertising. Not so.

For example, I tried a search on queries like "paleontology" and "pterodactyl" on a few different versions of Infospace’s metasearch. First, I used Webcrawler, which offers a version of Metacrawler results. The results I saw were mostly non-commercial in nature. In addition, Webcrawler has been relaunched as a "no banners, no buttons, no pop-ups" search site. The reduction in clutter is welcome, but some won’t be happy with the fact that some of the search results are sponsored listings not clearly demarcated from non-sponsored listings.

I tried the same search on Metacrawler and got a good overview of the topic. Results which appeared in all of the Inktomi, Google, and Teoma/Ask Jeeves indexes were ranked high on the page. There was only one sponsored result at the top – but of course this illustrates the fact that so much keyword inventory has still gone unsold to advertisers. Only one advertiser is paying for the keyword "pterodactyl," and only on Overture. A company called edinos is paying a mere one penny per click for this ad! I’d guess they’re doing well.

Metacrawler can do even better than this for an advanced user. Because we’ve been lulled into the above-mentioned media obsession with corporate wrongdoing, it’s been easy to assume that Metacrawler doesn’t offer its old strengths – but it does. If you want, for example, to customize your metasearch so that only Google and Teoma results appear, you can easily adjust and save your advanced settings. Or if you want to query all major directories and free-to-submit spider indexes plus all major paid inclusion indexes, but leave out sponsored listings from Overture and FindWhat, you can do this too. For me, the Teoma plus Google search was satisfying. It was a bit like just using Google, but coverage was slightly broader. Whenever you put several major indexes together, you’re likely to get additional coverage, which is why metasearch can offer an advantage over using a single engine like Google.

There is no question that the metasearch sector has faced big challenges. For one, there are seemingly fewer major indexes to include in a search. Excite Search, formerly a staple of most metasearch engines’ results, went the way of the dodo after the bankruptcy of Excite@Home. The brand was bought by Infospace, which discontinued the already-dead-anyway Excite Index and replaced the results with Overture sponsored listings. Shortly thereafter, Excite Search was relaunched as a metasearch engine, using results generated from a custom version of Metacrawler.

Essentially, then, Excite Search is now a more commercialized variant on Infospace’s flagship Metacrawler. Of course this means metasearch engines like Metacrawler (and, er, Excite Search) no longer have Excite Search to include in their results. The same goes for numerous other formerly vibrant search engines like Infoseek.

For those of you who still don't know what metasearch is (86% of consumers do not currently know what it is, but when it is explained to them, 84% find it "valuable"), it's a way of searching several different search engines or databases and presenting the results in a convenient format. The world leader in metasearch is Infospace, which has long been proprietor of Metacrawler. It also owns Dogpile, Webcrawler, and Excite Search. All four properties offer metasearch, presented in different formats and under different brand identities. One of the drawbacks of metasearch in the past couple of years has been an increasing tendency to stuff the results with sponsored listings, a trend which, confesses Richard Pelly, Infospace’s VP and General Manager, Search, contributed to the decline in public confidence in the search experience on Metacrawler and Dogpile. Although paid listings are still a significant part of Infospace’s overall mix, they’ve reversed course to some extent, making a conscious effort to reduce clutter and show advertising only where it’s relevant. Advanced searchers will never want to be buried under Dogpile’s avalanche of sponsored listings. But at least Infospace has thrown them a bone with a banner-free Webcrawler and advanced settings on Metacrawler.

York Baur, the company’s Executive VP of Wireline & Broadband Services, laments that the press "and Ralph Nader" have constructed an "artificial divide between a paid result and a ‘spidered’ result." Infospace believes that consumers should see relevant results whether they are paid or non-paid.

The technology behind Metacrawler has been beefed up partly in order to address search queries on either side of the "divide," leaving one to question how artificial this divide really is. The first thing Metacrawler does before returning results, explains Tasha Irvine, Infospace’s Product Manager, Search, is to decide whether a query is more likely to be commercial or non-commercial in nature based on large proprietary keyword lists. This determines the "mix" of paid versus non-paid results. If an inquiry is for "real estate in denver," more Overture and FindWhat results will be shown. If it’s for "paleontology," the number sponsored listings will be less, and they’ll be shown further down the page.

Much has been made of the technological breakthroughs and "under the hood" computing power that has allowed spidering engines like Inktomi and Google to outdo their contemporaries. Irvine makes it clear that Metacrawler’s lead over competing metasearch engines is also partly buttressed by high-powered technology. Speed is a huge issue for metasearch engines; querying multiple databases and returning aggregated results takes time. And when users press their browser’s "back" button, typically the same page needs to be generated and it may not be from a cache. Slow metasearch services, therefore, aren’t very user-friendly.

Contractual relationships also set Infospace’s metasearch engines apart from the competition. According to Infospace’s Baur, Infospace is the only metasearch provider that has a contractual arrangement with Google to include the Google index in its metasearch. "There have been questions about whether metasearch is legal," says Baur. "Our view is, as with any grey area relating to Internet copyright, it depends on how you do it. We have signed agreements with several major search companies." This special access not only means that Infospace’s metasearch is authorized; it also allows it to work faster and more reliably.

The decline of early-adopters’ interest in metasearch – and if that trend continued, its potential extinction – has has been fueled not only by an excess of paid results in the mix, but also by the erosion of the former raison-d’être for metasearch: the premise that a number of distinct, vibrant, non-paid web search indexes exist and that metasearch can "query them all" to save time and to help in comparisons. The dominance of Google has led many consumers to assume that Google is all they need; in some way, that Google is search much as eBay is auctions and Amazon is books. Few consumers do a "meta-book-store" search. Little is heard anymore about services like AuctionRover which used to query several auction sites at once. Could a similar phenomenon be happening in the search space now that Google is ascendant?

According to Irvine, Metacrawler has had to change constantly to reflect the state of the search industry, and it will continue to do so. In recent months, it has added FAST Search, changed its relationship with Ask Jeeves, and, reluctantly, removed Wisenut. "We have a number of tests that we require our search partners to pass," says Irvine. "One criterion is that they can handle the high volume of queries we send them. Wisenut, when we first partnered with them, was a small company and got overwhelmed. Their systems couldn’t handle the load." In short, Infospace is always re-evaluating the mix of search results that go into its metasearch.

When Google’s halo effect wears off, the current environment of "single engine dominance" may change again. Metasearch could catch on again as consumers realize that one search engine can’t do everything for them.

The re-emergence of metasearch depends in large part on search enthusiasts finding it useful again. Tens of thousands of university librarians, elementary school teachers, and other research experts have a big influence on the adoption of cool tools by students seeking information to write papers, and ultimately, to perform other types of job-related and lifestyle-related searching. Thus the importance of managing the delicate balance amongst different forms of paid and non-paid listings: experts won’t recommend a tool if it’s too blatantly commercialized.

Stories claiming a 19th-century sighting of the once-mighty pterodactyl were probably hoaxes. Metasearch, on the other hand, has merely been hibernating but remains very much alive. As long as metasearch product developers have the genetic code of a variety of ingenious and powerful search tools at their disposal, it seems inevitable that future generations of metasearch will fly high.