. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 5 Solving specific problems 131 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . crawler using a web service. 130 Chapter 4. Built-in services CHAPTER 5 Solving specific problems 5.1 Frequently Asked Questions 5.1.1 How does Scrapy compare to BeautifulSoup or lxml? BeautifulSoup py file you can run it with: scrapy runspider my_spider.py See runspider command for more info. 5.1. Frequently Asked Questions 133 Scrapy Documentation, Release 1.5.2 5.1.16 I get “Filtered offsite

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 5 Solving specific problems 137 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . crawler using a web service. 136 Chapter 4. Built-in services CHAPTER 5 Solving specific problems 5.1 Frequently Asked Questions 5.1.1 How does Scrapy compare to BeautifulSoup or lxml? BeautifulSoup py file you can run it with: scrapy runspider my_spider.py See runspider command for more info. 5.1. Frequently Asked Questions 139 Scrapy Documentation, Release 1.6.0 5.1.16 I get “Filtered offsite

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5 Solving specific problems 127 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . requests to domains outside the ones covered by the spider. For more info see: OffsiteMiddleware. 5.1. Frequently Asked Questions 129 Scrapy Documentation, Release 1.2.3 What is the recommended way to XPath selector doesn’t return any items You may need to remove namespaces. See Removing namespaces. 5.1. Frequently Asked Questions 131 Scrapy Documentation, Release 1.2.3 Debugging Spiders This document

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5 Solving specific problems 121 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . requests to domains outside the ones covered by the spider. For more info see: OffsiteMiddleware. 5.1. Frequently Asked Questions 123 Scrapy Documentation, Release 1.1.3 What is the recommended way to XPath selector doesn’t return any items You may need to remove namespaces. See Removing namespaces. 5.1. Frequently Asked Questions 125 Scrapy Documentation, Release 1.1.3 Debugging Spiders This document

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5 Solving specific problems 129 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . requests to domains outside the ones covered by the spider. For more info see: OffsiteMiddleware. 5.1. Frequently Asked Questions 131 Scrapy Documentation, Release 1.3.3 What is the recommended way to XPath selector doesn’t return any items You may need to remove namespaces. See Removing namespaces. 5.1. Frequently Asked Questions 133 Scrapy Documentation, Release 1.3.3 Debugging Spiders This document

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 5 Solving specific problems 131 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . requests to domains outside the ones covered by the spider. For more info see: OffsiteMiddleware. 5.1. Frequently Asked Questions 133 Scrapy Documentation, Release 1.4.0 What is the recommended way to XPath selector doesn’t return any items You may need to remove namespaces. See Removing namespaces. 5.1. Frequently Asked Questions 135 Scrapy Documentation, Release 1.4.0 Debugging Spiders This document

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 5 Solving specific problems 139 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . crawler using a web service. 138 Chapter 4. Built-in services CHAPTER 5 Solving specific problems 5.1 Frequently Asked Questions 5.1.1 How does Scrapy compare to BeautifulSoup or lxml? BeautifulSoup runspider command. For example, if you have a spider written in a my_spider.py file you can run it with: 5.1. Frequently Asked Questions 141 Scrapy Documentation, Release 1.7.4 scrapy runspider my_spider.py

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 5 Solving specific problems 151 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . using a web service. 150 Chapter 4. Built-in services CHAPTER FIVE SOLVING SPECIFIC PROBLEMS 5.1 Frequently Asked Questions 5.1.1 How does Scrapy compare to BeautifulSoup or lxml? BeautifulSoup DEFAULT_REQUEST_HEADERS setting. 5.1.14 Where can I find some example Scrapy projects? See Examples. 5.1. Frequently Asked Questions 153 Scrapy Documentation, Release 1.8.4 5.1.15 Can I run a spider without

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5 Solving specific problems 117 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . download delay of 2 (or higher) in your spider: class MySpider(CrawlSpider): name = 'myspider' 5.1. Frequently Asked Questions 119 Scrapy Documentation, Release 1.0.7 download_delay = 2 # [ ... rest 0.7 ... process = CrawlerProcess({ 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)' }) process.crawl(MySpider) process.start() # the script will block here until the crawling is finished

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5 Solving specific problems 79 5.1 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Documentation, Release 0.12.0 78 Chapter 4. Built-in services CHAPTER 5 Solving specific problems 5.1 Frequently Asked Questions 5.1.1 How does Scrapy compare to BeautifulSoul or lxml? BeautifulSoup code ... ] Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting. 5.1. Frequently Asked Questions 81 Scrapy Documentation, Release 0.12.0 5.1.20 Can I call pdb.set_trace()