how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 5.5.2 Reduce log level When doing broad crawls any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should 5.5. Broad Crawls 87 Scrapy Documentation, Release 0.16

how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Reduce log level When doing broad crawls you any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls

processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load. In addition to dispatching and mulitplied by the value in max_proc_per_cpu option. Defaults to 0. max_proc_per_cpu The maximum number of concurrent Scrapy process that will be started per cpu. Defaults to 4. debug Whether debug mode eggs_dir = eggs logs_dir = logs logs_to_keep = 5 dbs_dir = dbs max_proc = 0 max_proc_per_cpu = 4 http_port = 6800 debug = off runner = scrapyd.runner application = scrapyd.app.application

processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load. In addition to dispatching and mulitplied by the value in max_proc_per_cpu option. Defaults to 0. 98 Chapter 5. Solving specific problems Scrapy Documentation, Release 0.12.0 max_proc_per_cpu The maximum number of concurrent Scrapy Scrapy process that will be started per cpu. Defaults to 4. debug Whether debug mode is enabled. Defaults to off. When debug mode is enabled the full Python traceback will be returned (as plain text responses)

processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load. In addition to dispatching and mulitplied by the value in max_proc_per_cpu option. Defaults to 0. max_proc_per_cpu The maximum number of concurrent Scrapy process that will be started per cpu. Defaults to 4. debug Whether debug mode eggs_dir = eggs logs_dir = logs logs_to_keep = 5 dbs_dir = dbs max_proc = 0 max_proc_per_cpu = 4 http_port = 6800 debug = off runner = scrapyd.runner application = scrapyd.app.application

processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load. In addition to dispatching and mulitplied by the value in max_proc_per_cpu option. Defaults to 0. max_proc_per_cpu The maximum number of concurrent Scrapy process that will be started per cpu. Defaults to 4. debug Whether debug mode [scrapyd] eggs_dir = eggs logs_dir = logs logs_to_keep = 5 dbs_dir = dbs max_proc = 0 max_proc_per_cpu = 4 http_port = 6800 debug = off runner = scrapyd.runner application = scrapyd.app.application 5

how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 5.5.2 Reduce log level When doing broad crawls any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls

how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 5.5.2 Reduce log level When doing broad crawls any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls

how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 5.5.2 Reduce log level When doing broad crawls any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls

how much CPU you crawler will have available. A good starting point is 100, but the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS = 100 Reduce log level When doing broad crawls you any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls