Web Crawling › Key Parameters
Michael BrenndoerferJanuary 8, 20261 min read
Language AI HandbookAll chapters Part 20 of 67Data Curation
Web Crawling › Key ParametersOne section · Sections in this chapter7 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The main parameters governing a web crawler's behavior are:
- crawl_delay: Minimum seconds between requests to the same host. Typical values range from 0.5 to 5 seconds. Always honor the
Crawl-delaydirective in robots.txt when it specifies a higher value. - max_depth: Maximum link distance from seed URLs to follow. Shallow crawls (depth 2-3) are efficient and produce higher average quality; deep crawls (depth 5+) surface long-tail content but bring in more noise.
- max_pages: A stopping condition based on page count. Combined with max_depth, it prevents runaway crawls.
- keywords (focused crawling): Relevance keywords used to score and prioritize URLs. More specific keywords produce higher-precision focused crawls at the cost of recall.
- user_agent: The crawler's identifier string. Always include contact information so site operators can reach you.