are crawling. 5.12.1 Design goals 1. be nicer to sites instead of using default download delay of zero 5.10. Ubuntu packages 109 Scrapy Documentation, Release 0.22.0 2. automatically adjust scrapy to Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire. Changed in version 0.11: Before 0.11, zero meant cached requests always expire. HTTPCACHE_DIR Default: documentation for more info): •DEPTH_LIMIT - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed. •DEPTH_STATS - Whether to collect depth stats. •DEPTH_PRIORITY - Whether

are crawling. 5.12.1 Design goals 1. be nicer to sites instead of using default download delay of zero 2. automatically adjust scrapy to the optimum crawling speed, so the user doesn’t have to tune the Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire. Changed in version 0.11: Before 0.11, zero meant cached requests always expire. HTTPCACHE_DIR Default: documentation for more info): •DEPTH_LIMIT - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed. •DEPTH_STATS - Whether to collect depth stats. •DEPTH_PRIORITY - Whether

website you are crawling. Design goals 1. be nicer to sites instead of using default download delay of zero 2. automatically adjust scrapy to the optimum crawling speed, so the user doesn’t have to tune the Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire. Changed in version 0.11: Before 0.11, zero meant cached requests always expire. HTTPCACHE_DIR Default: documentation for more info): DEPTH_LIMIT - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed. DEPTH_STATS - Whether to collect depth stats. DEPTH_PRIORITY - Whether

the first clickable element. In addition to html attributes, the control can be identified by its zero-based index relative to other submittable inputs inside the form, via the nr attribute. • dont_click number of concurrent (ie. simultaneous) requests that will be performed to any single IP. If non- zero, the CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words IP, not per domain. This setting also affects DOWNLOAD_DELAY: if CONCURRENT_REQUESTS_PER_IP is non-zero, download delay is enforced per IP, not per domain. DEFAULT_ITEM_CLASS Default: 'scrapy.item.Item'

website you are crawling. Design goals 1. be nicer to sites instead of using default download delay of zero 2. automatically adjust scrapy to the optimum crawling speed, so the user doesn’t have to tune the Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire. Changed in version 0.11: Before 0.11, zero meant cached requests always expire. HTTPCACHE_DIR Default: documentation for more info): DEPTH_LIMIT - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed. DEPTH_STATS - Whether to collect depth stats. DEPTH_PRIORITY - Whether

are crawling. 5.12.1 Design goals 1. be nicer to sites instead of using default download delay of zero 2. automatically adjust scrapy to the optimum crawling speed, so the user doesn’t have to tune the Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire. Changed in version 0.11: Before 0.11, zero meant cached requests always expire. HTTPCACHE_DIR Default: documentation for more info): •DEPTH_LIMIT - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed. •DEPTH_STATS - Whether to collect depth stats. •DEPTH_PRIORITY - Whether

are crawling. 5.12.1 Design goals 1. be nicer to sites instead of using default download delay of zero 5.10. Ubuntu packages 107 Scrapy Documentation, Release 0.20.2 2. automatically adjust scrapy to Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire. Changed in version 0.11: Before 0.11, zero meant cached requests always expire. HTTPCACHE_DIR Default: documentation for more info): •DEPTH_LIMIT - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed. •DEPTH_STATS - Whether to collect depth stats. •DEPTH_PRIORITY - Whether

the first clickable element. In addition to html attributes, the control can be identified by its zero-based index relative to other submittable inputs inside the form, via the nr attribute. • dont_click number of concurrent (ie. simultaneous) requests that will be performed to any single IP. If non- zero, the CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words setting also affects DOWNLOAD_DELAY and AutoThrottle extension: if CONCURRENT_REQUESTS_PER_IP is non-zero, download delay is enforced per IP, not per domain. DEFAULT_ITEM_CLASS Default: 'scrapy.item.Item'

the first clickable element. In addition to html attributes, the control can be identified by its zero-based index relative to other submittable inputs inside the form, via the nr attribute. • dont_click number of concurrent (ie. simultaneous) requests that will be performed to any single IP. If non- zero, the CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words setting also affects DOWNLOAD_DELAY and AutoThrottle extension: if CONCURRENT_REQUESTS_PER_IP is non-zero, download delay is enforced per IP, not per domain. DEFAULT_ITEM_CLASS Default: 'scrapy.item.Item'

the first clickable element. In addition to html attributes, the control can be identified by its zero-based index relative to other submittable inputs inside the form, via the nr attribute. dont_click maximum number of concurrent (ie. simultaneous) requests that will be performed to any single IP. If non-zero, the CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words not per domain. This setting also affects DOWNLOAD_DELAY: if CONCURRENT_REQUESTS_PER_IP is non- zero, download delay is enforced per IP, not per domain. DEFAULT_ITEM_CLASS Default: 'scrapy.item.Item'