Web Crawling › Robots.txt and Ethical Crawling

Web crawlers have enormous power. A single aggressive crawler can bring down a web server by flooding it with requests. Even well-intentioned crawlers can cause problems: generating excessive server load, consuming bandwidth that costs site owners money, and accessing private or sensitive content that was never meant to be indexed.

The robots exclusion protocol, formalized as robots.txt, is the web's social contract between site owners and crawlers. When a crawler first visits a domain, it should fetch https://example.com/robots.txt and parse the access rules before crawling any other URL on that domain. This file tells the crawler which paths are off limits, which crawlers are welcome, and how fast requests should be made.

The robots.txt format specifies rules for different user agents using User-agent directives, followed by Disallow and Allow rules. The wildcard * matches any user agent not explicitly named.

In this example, all crawlers are blocked from /private/ and /admin/, but explicitly allowed to access /public/. Googlebot has an additional exclusion for /staging/ and must wait at least 1 second between requests. The sitemap declaration points to a structured list of URLs the site owner wants crawled.

Robots.txt parsing has several edge cases that matter for correct implementation. The Allow directive takes precedence over Disallow when both match a URL, but the more specific rule wins when there is no explicit precedence conflict. An Allow: /public/ rule overrides Disallow: / for any URL under /public/. The order of rules within a group matters in some implementations, while others apply longest-match semantics. Wildcards (*) and end-of-string anchors ($) add expressiveness: Disallow: /*.pdf$ blocks all PDF files regardless of directory.

The Crawl-delay directive specifies the minimum number of seconds to wait between successive requests to the server. Respecting this directive is polite and often legally required in jurisdictions where unauthorized automated access may violate terms of service. When both the robots.txt crawl delay and your own configured delay exist, you should honor whichever is more conservative.

The Sitemap directive points to XML sitemap files that enumerate the site's publicly available pages. Smart crawlers fetch these sitemaps to seed their frontier with known good URLs rather than relying entirely on link following. A well-maintained sitemap tells the crawler exactly which URLs the site owner considers canonical and worth indexing, eliminating guesswork about URL normalization and reducing the chance of crawling duplicate or unwanted pages.

Respecting robots.txt is both an ethical obligation and a practical necessity. Many websites disallow crawling of user-generated content areas, login-protected pages, or dynamically generated search results that would produce near-infinite URLs. Crawling these areas wastes resources and produces low-quality data. Pages behind authentication walls return redirect responses or login forms rather than actual content. Search result pages produce query-dependent content that changes with every request and may contain little informational value as training data.

The legal dimension is significant. Some jurisdictions treat ignoring robots.txt as unauthorized computer access. In the United States, the Computer Fraud and Abuse Act has been used to argue against unauthorized scraping, though legal interpretations vary. In Europe, the GDPR creates obligations around processing personal data, which web pages may contain. Beyond legality, violating robots.txt damages trust between the open-web ecosystem and the research community. Sites that feel violated by crawlers often block entire IP ranges, penalizing legitimate crawlers along with bad actors.

Some data pipelines used to train language models have been criticized for ignoring robots.txt at scale. When Common Crawl began operations, some site owners found their content in crawl data despite having disallowed bots. This tension between the value of open training data and the rights of content creators has become a central legal and ethical debate in AI development. If you are building a crawler, respecting robots.txt is the minimum acceptable standard.

Even when robots.txt grants access, good crawlers implement additional politeness measures. The most important is rate limiting. Sending too many requests too quickly stresses servers and degrades the experience for human visitors. A common heuristic is to wait a multiple of the server's response time before issuing the next request, adapting automatically to server load: if a server takes 2 seconds to respond, wait an additional 4 seconds before sending the next request. This adaptive delay naturally backs off when servers are under load without requiring you to hard-code domain-specific delays.

IP-level rate limiting matters too. A single website may be hosted behind a shared IP that serves thousands of domains. Hitting that IP with aggregate traffic from all those domains can cause collateral damage to unrelated sites. Production crawlers track per-host request rates and per-IP request rates to avoid this kind of collateral congestion.

Identifying yourself through the User-Agent HTTP header is another courtesy. Well-behaved crawlers provide a user agent string that includes a URL or email address where the site owner can contact you if your crawler causes problems. This allows site operators to reach out about issues rather than simply blocking your IP. Compare Mozilla/5.0 (compatible; MyCrawler/1.0) with MyCrawler/1.0 (+https://myproject.org/crawler; contact@myproject.org). The second form gives site operators actionable information.

Crawlers should also implement exponential backoff when they encounter server errors. A 503 Service Unavailable response means the server is temporarily overwhelmed. Immediately retrying adds to that overwhelm. Instead, wait an exponentially increasing delay: 30 seconds, then 60, then 120, then give up. This pattern is respectful and also more likely to succeed, since transient overloads typically resolve within minutes.