and HTML parser • parsel, an HTML/XML data extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP header will not be sent at all. • cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). • body (str) – the response body. It must be str, not unicode, unless you’re using

and HTML parser • parsel, an HTML/XML data extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP header will not be sent at all. • cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). • body (str) – the response body. It must be str, not unicode, unless you’re using

and HTML parser • parsel, an HTML/XML data extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP header will not be sent at all. • cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). • body (bytes) – the response body. To access the decoded text as str (unicode in

and HTML parser • parsel, an HTML/XML data extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP header will not be sent at all. • cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). • body (bytes) – the response body. To access the decoded text as str (unicode in

HTML/XML data extraction library written on top of lxml, w3lib [https://pypi.python.org/pypi/w3lib], a multi-purpose helper for dealing with URLs and web page encodings twisted [https://twistedmatrix.com/], headers of this request. The dict values can be strings (for single valued headers) or lists (for multi- valued headers). If None is passed as value, the HTTP header will not be sent at all. cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi- valued headers). body (str) – the response body. It must be str, not unicode, unless you’re using

HTML/XML data extraction library written on top of lxml, w3lib [https://pypi.python.org/pypi/w3lib], a multi-purpose helper for dealing with URLs and web page encodings twisted [https://twistedmatrix.com/], headers of this request. The dict values can be strings (for single valued headers) or lists (for multi- valued headers). If None is passed as value, the HTTP header will not be sent at all. cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi- valued headers). body (str) – the response body. It must be str, not unicode, unless you’re using

and HTML parser • parsel, an HTML/XML data extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP header will not be sent at all. • cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). • body (str) – the response body. It must be str, not unicode, unless you’re using

HTML/XML data extraction library written on top of lxml, w3lib [https://pypi.python.org/pypi/w3lib], a multi-purpose helper for dealing with URLs and web page encodings twisted [https://twistedmatrix.com/], headers of this request. The dict values can be strings (for single valued headers) or lists (for multi- valued headers). If None is passed as value, the HTTP header will not be sent at all. cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi- valued headers). body (str) – the response body. It must be str, not unicode, unless you’re using

headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP header will not be sent at all. • cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). • status (integer) – the HTTP status of the response. Defaults to 200. • body (str) REACTOR_THREADPOOL_MAXSIZE Default: 10 The maximum limit for Twisted Reactor thread pool size. This is common multi-purpose thread pool used by various Scrapy components. Threaded DNS Resolver, BlockingFeedStorage

Distributed crawls Scrapy doesn’t provide any built-in facility for running crawls in a distribute (multi-server) manner. However, there are some ways to distribute crawls, which vary depending on how you headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP header will not be sent at all. • cookies (dict headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). • status (integer) – the HTTP status of the response. Defaults to 200. • body (str)