Google bots get the red carpet treatment

posted by   theblackbull (133)   9 months ago (technospider.blogspot.com)

Webmasters who control automated web-crawler access to their sites using 'robots.txt' files have a bias that favours Google over other search engines, according to new research. The claim was made by researchers at Penn State University based on the results of a study of more than 7,500 websites. C. Lee Giles, David Reese professor of Information Sciences and Technology at Penn State, who led the research team which developed the BotSeer search engine for the study, described the pro-Google bias as "surprising". "We expected that 'robots.txt' files would treat all search engines equally, or maybe disfavour certain obnoxious bots," he said. "So we were surprised to discover a strong correlation between the favoured robots and search engine market share."