Back to Directory

Search Engines  >> Directories >> Pay Per Click >> Meta Search >> Robots

---

All About Search Indexing Robots and Spiders - Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.

GoogleStats - A free Open Source PHP application tracking the visits of Googlebot and many more robots on your site and giving full statistics in tables and graphics.

List of Robot Agent Strings - A list from PGTS of Web robots with the identifying data they leave in Web site logs.

Psychedelix List of User-Agents - Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.

Robots, Spiders and Other User Agents: a Resource for WebMasters - José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web.

Robots.txt syntax checker - Online tool for syntax verification to robots.txt files.

Search Engine IP Addresses - Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.

Search Engine Robots and Other User Agents - John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.

Search Engine Spider Identification - Webmaster World forum for identifying search engine robots.

The User-Agent Database - Icehouse has gathered a list of spiders (robots or crawlers), browsers and harvesters, sorted into a searchable database.

The Web Robots Pages - Martijn Koster's FAQs about the programmes that index the Web, how to direct visiting robots, a database of robots, an archived mailing list, background reading, and related links.