. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.5 response.headers: raise ContractFail('X-CustomHeader not present') Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often crawler import CrawlerProcess class MySpider(scrapy.Spider): # Your spider definition 5.4. Common Practices 125 Scrapy Documentation, Release 1.0.7 ... process = CrawlerProcess({ 'USER_AGENT': 'Mozilla/4

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.5 response.headers: raise ContractFail(’X-CustomHeader not present’) 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often Twisted asynchronous networking library, so you need run it inside the Twisted reactor. 5.4. Common Practices 91 Scrapy Documentation, Release 0.22.0 Note that you will also have to shutdown the Twisted reactor

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.5 response.headers: raise ContractFail(’X-CustomHeader not present’) 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often Twisted asynchronous networking library, so you need run it inside the Twisted reactor. 5.4. Common Practices 89 Scrapy Documentation, Release 0.20.2 Note that you will also have to shutdown the Twisted reactor

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 5.5 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often fall into any other specific section. 5.4. Common Practices 161 Scrapy MySpider1(scrapy.Spider): # Your first spider definition ... (continues on next page) 5.4. Common Practices 163 Scrapy Documentation, Release 1.8.4 (continued from previous page) class MySpider2(scrapy

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 5.5 response.headers: raise ContractFail('X-CustomHeader not present') 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often MySpider2(scrapy.Spider): # Your second spider definition (continues on next page) 5.4. Common Practices 141 Scrapy Documentation, Release 1.5.2 (continued from previous page) ... process = CrawlerProcess()

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 5.5 response.headers: raise ContractFail('X-CustomHeader not present') 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often first spider definition ... class MySpider2(scrapy.Spider): (continues on next page) 5.4. Common Practices 147 Scrapy Documentation, Release 1.6.0 (continued from previous page) # Your second spider definition

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 5.5 get("SCRAPY_CHECK"): pass # Do some scraper adjustments when a check is running 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often definition ... process = CrawlerProcess( settings={ "FEEDS": { (continues on next page) 5.4. Common Practices 189 Scrapy Documentation, Release 2.11.1 (continued from previous page) "items.json": {"format":

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 5.5 get("SCRAPY_CHECK"): pass # Do some scraper adjustments when a check is running 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often CrawlerRunner from scrapy.utils.log import configure_logging (continues on next page) 5.4. Common Practices 189 Scrapy Documentation, Release 2.10.1 (continued from previous page) class MySpider(scrapy

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 5.5 get('SCRAPY_CHECK'): pass # Do some scraper adjustments when a check is running 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often script will block here until the crawling is finished See also: Reactor Overview 5.4. Common Practices 183 Scrapy Documentation, Release 2.7.1 5.4.2 Running multiple spiders in the same process By

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.5 get("SCRAPY_CHECK"): pass # Do some scraper adjustments when a check is running 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often CrawlerRunner from scrapy.utils.log import configure_logging (continues on next page) 5.4. Common Practices 187 Scrapy Documentation, Release 2.9.0 (continued from previous page) class MySpider(scrapy