CSIRO has been developing methods to compare search engine performance so users can choose the best engine for their needs.

Dr David Hawking of CSIRO's Electronic Content Technologies group says that search engine performance varies according to the task.

"Some search engines are very good at finding lots of 'hits' for a query, but may fall down in how they rank the results. The most relevant sites can end up too far down the list to be noticed," he says.

"Poor ranking wastes people's time by forcing them to scan through irrelevant material. In the workplace that translates to lost productivity.

"So one thing we are looking at is how good various engines are at putting relevant results on the first page."

In a recent test some of the top performers at producing highly ranked and relevant hits were Google (www.google.com), Fast (www.alltheweb.com) and Northern Light (www.northernlight.com).

"The differences between the top three were not statistically significant so we have not put them in any order," says Dr Hawking.

People use search engines for various reasons. They might be looking for information on a topic but they might also be looking for a site offering a particular service.

"If you want to order flowers online, you don't want a whole lot of 'how to' articles about running an online flower shop," says Dr Hawking.

"So we selected some actual natural language queries from search engine logs such as 'Where can I order flowers online?' to see how good each search engine was at finding a service rather than just information.

When it came to locating services, Google, Northern Light and Fast were again the top performers. CSIRO's is the first published study of how well search engines perform at finding online services.

A third way people commonly use search engines is to locate the home page of a particular organisation. The CSIRO study looked at how each search engine performed in locating the URL for an airline's homepage, using their name (as listed in the International Air Transport Association members list) as the query.

The top three in the home page location test were Fast, Google and Microsoft (search.msn.com).

"Ranking ability is particularly relevant in this type of query. Some search engines can find the right page - but do not rank it highly enough. Finding home pages clearly requires specialised searching techniques and not all engines have successfully implemented these," says Dr Hawking.

The evaluation methods are being developed to ensure a fair comparison of search engines. Because performance can vary over time, all tests of a particular type are done on the same day. The same sets of queries are submitted to all search engines and an independent assessor evaluates all the results.

Dr Hawking says the work is intended to be of value both to the Internet community as a whole, for selecting the best search engines for a particular job, and to the search engine operators themselves in seeing how their product compares with the competition.

"It's important to remember that search engine operators are making improvements all the time, so the performance we measure on a particular day may not be what a user will find on a different day," he says.

"We'd like to be in a position to publish regular evaluations on the major search engines as a service to Internet users and to encourage improvements across the industry. We are currently seeking support from the search engine operators to do this."

More Information:

Dr David Hawking, CSIRO, 02 6216 7060, 0409 031 007, david.hawking@csiro.au

Media assistance:

Tom McGinness, CSIRO, 02 9325 3227, 0419 419 210  tom.mcginness@csiro.au