result is cached after the first call, so you can access response.text multiple times without extra overhead. Note: unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Increase Twisted IO thread pool maximum size

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note: unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Increase Twisted IO thread pool maximum size

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note: unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Increase Twisted IO thread pool maximum size

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Increase Twisted IO thread pool maximum size

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note: unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 5.5.2 Increase Twisted IO thread pool maximum

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note: unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 5.5.2 Increase Twisted IO thread pool maximum

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Increase Twisted IO thread pool maximum size

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Increase Twisted IO thread pool maximum size

result is cached after the first call, so you can access response.text multiple times without extra overhead. Note: unicode(response.body) is not a correct way to convert response body to unicode: you would how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Increase Twisted IO thread pool maximum size

how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 5.5.2 Reduce log level When doing broad crawls any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls