example: [settings] default = myproject.settings By default, Scrapy projects use a SQLite [http://en.wikipedia.org/wiki/SQLite] database to store persistent runtime data of the project, such as the spider queue queue (the list of spiders that are scheduled to run). By default, this SQLite database is stored in the project data directory which, by default, is the .scrapy directory inside the project root directory usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector

Scrapy projects use a SQLite database to store persistent runtime data of the project, such as the spider queue (the list of spiders that are scheduled to run). By default, this SQLite database is stored usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector is a built-in Scrapy extension which comes enabled by default, but you can also disable it if you want. For more information about the extension itself see Telnet console extension. 4.4.1 How to access

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 i 5.12 AutoThrottle extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.13 Jobs: usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats): self.stats

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 i 5.12 AutoThrottle extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.13 Benchmarking usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats): self.stats

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 i 5.12 AutoThrottle extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5.13 Benchmarking usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats): self.stats

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.12 AutoThrottle extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.13 Benchmarking usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats): self.stats

on Ubuntu Scrapy Service (scrapyd) Deploying your Scrapy project in production. AutoThrottle extension Adjust crawl rate dynamically based on load. Jobs: pausing and resuming crawls Learn how to pause usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats):

packages easily on Ubuntu Scrapyd Deploying your Scrapy project in production. AutoThrottle extension Adjust crawl rate dynamically based on load. Benchmarking Check how Scrapy performs on your hardware usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats):

packages easily on Ubuntu Scrapyd Deploying your Scrapy project in production. AutoThrottle extension Adjust crawl rate dynamically based on load. Benchmarking Check how Scrapy performs on your hardware usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats):

packages easily on Ubuntu Scrapyd Deploying your Scrapy project in production. AutoThrottle extension Adjust crawl rate dynamically based on load. Benchmarking Check how Scrapy performs on your hardware usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from. Another feature of the Stats Collector Collector uses Access the stats collector through the stats attribute. Here is an example of an extension that access stats: class ExtensionThatAccessStats(object): def __init__(self, stats):