Scrapy Documentation Release 1.0.7 Scrapy developers Aug 14, 2017 First steps 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue that prevents xpath('//title/text()').re('(\w+):') Out[5]: [u'Computers', u'Programming', u'Languages', u'Python'] 14 Chapter 2. First steps Scrapy Documentation, Release 1.0.7 Extracting the data Now, let’s try to

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue that prevents for brevity) 2016-09-20 14:48:00 [scrapy] INFO: Spider opened 2016-09-20 14:48:00 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items ˓→(at 0 items/min) 2016-09-20 14:48:00 [scrapy] DEBUG: console listening on 127.0.0.1:6023 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (404) (referer: None) 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (200)

0 码力 | 266 页 | 1.10 MB | 1 年前

3

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue that prevents for brevity) 2016-09-20 14:48:00 [scrapy] INFO: Spider opened 2016-09-20 14:48:00 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items ˓→(at 0 items/min) 2016-09-20 14:48:00 [scrapy] DEBUG: console listening on 127.0.0.1:6023 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (404) (referer: None) 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (200)

0 码力 | 260 页 | 1.12 MB | 1 年前

3

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue [https://github all stats: >>> stats.get_stats() {'pages_crawled': 1238, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} Available Stats Collectors Besides the basic StatsCollector there are other language? Try changing the default Accept-Language [http://www.w3.org/Protocols/rfc2616/rfc2616- sec14.html#sec14.4] request header by overriding the DEFAULT_REQUEST_HEADERS setting. Where can I find some

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue that prevents and extract_first() methods, you can also use the re() method to extract using regular expressions: 14 Chapter 2. First steps Scrapy Documentation, Release 1.3.3 >>> response.css('title::text').re(r'Quotes Get all stats: >>> stats.get_stats() {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} Available Stats Collectors Besides the basic StatsCollector there are other

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue [https://github for brevity) 2016-09-20 14:48:00 [scrapy] INFO: Spider opened 2016-09-20 14:48:00 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-09-20 14:48:00 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (404) (referer: None) 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (200)

0 码力 | 322 页 | 582.29 KB | 1 年前

3

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue [https://github for brevity) 2016-09-20 14:48:00 [scrapy] INFO: Spider opened 2016-09-20 14:48:00 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-09-20 14:48:00 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (404) (referer: None) 2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (200)

0 码力 | 330 页 | 548.25 KB | 1 年前

3

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue [https://github Get all stats: >>> stats.get_stats() {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} Available Stats Collectors Besides the basic StatsCollector there are other language? Try changing the default Accept-Language [https://www.w3.org/Protocols/rfc2616/rfc2616- sec14.html#sec14.4] request header by overriding the DEFAULT_REQUEST_HEADERS setting. Where can I find some

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue that prevents instance of SelectorList. When you know you just want the first result, as in this case, you can do: 14 Chapter 2. First steps Scrapy Documentation, Release 1.5.2 >>> response.css('title::text').extract_first() Get all stats: >>> stats.get_stats() {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} 4.2.2 Available Stats Collectors Besides the basic StatsCollector there are

headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run: xcode-select --install There’s a known issue that prevents Selector, etc) [s] crawler (continues on next page) 14 Chapter 2. First steps Scrapy Documentation, Release 1.6.0 (continued from previous page) [s] item Get all stats: >>> stats.get_stats() {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} 4.2.2 Available Stats Collectors Besides the basic StatsCollector there are