Learn about bleeding-edge features. © Copyright 2008-2013, Scrapy developers. Last updated on Apr 07, 2016. Created using Sphinx 1.3.5. index modules | next | previous | Scrapy 0.24.6 documentation org/community/]. Thanks for your interest! © Copyright 2008-2013, Scrapy developers. Last updated on Apr 07, 2016. Created using Sphinx 1.3.5. index modules | next | previous | Scrapy 0.24.6 documentation AUR Scrapy package: yaourt -S scrapy © Copyright 2008-2013, Scrapy developers. Last updated on Apr 07, 2016. Created using Sphinx 1.3.5. index modules | next | previous | Scrapy 0.24.6 documentation

to this: 2014-01-23 18:13:07-0400 [scrapy] INFO: Scrapy started (bot: tutorial) 2014-01-23 18:13:07-0400 [scrapy] INFO: Optional features available: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Overridden settings: {} 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled extensions: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled Enabled spider middlewares: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled item pipelines: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Spider opened 2014-01-23 18:13:08-0400 [scrapy] DEBUG: Crawled (200)

Scrapy Documentation Release 0.24.6 Scrapy developers April 07, 2016 Contents 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . to this: 2014-01-23 18:13:07-0400 [scrapy] INFO: Scrapy started (bot: tutorial) 2014-01-23 18:13:07-0400 [scrapy] INFO: Optional features available: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Overridden settings: {} 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled extensions: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled spider

to this: 2014-01-23 18:13:07-0400 [scrapy] INFO: Scrapy started (bot: tutorial) 2014-01-23 18:13:07-0400 [scrapy] INFO: Optional features available: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Overridden settings: {} 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled extensions: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled spider spider middlewares: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled item pipelines: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Spider opened 2014-01-23 18:13:08-0400 [scrapy] DEBUG: Crawled (200)

Some of the tips are based on this post from ScrapingHub’s blog [https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/]. Working with relative XPaths Keep in mind that if Selector(text='') >>> sel.css('.shout').xpath('./time/@datetime').getall() ['2014-07-23 19:00'] This is cleaner than using the verbose Whose SHA1 hash is: 3afec3b4765f8f0a07b78f98c07b83f013567a0a Will be downloaded and stored in the following file: /full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg Where:

Some of the tips are based on this post from ScrapingHub’s blog [https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping- trenches/]. Working with relative XPaths Keep in mind that if Selector(text='') >>> sel.css('.shout').xpath('./time/@datetime').getall() ['2014-07-23 19:00'] This is cleaner than using the verbose Whose SHA1 hash is: 3afec3b4765f8f0a07b78f98c07b83f013567a0a Will be downloaded and stored in the following file: /full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg Where:

Selector(text='') >>> sel.css('.shout').xpath('./time/@datetime').getall() ['2014-07-23 19:00'] This is cleaner than using the verbose Whose SHA1 hash is: 3afec3b4765f8f0a07b78f98c07b83f013567a0a Will be downloaded and stored in the following file: /full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg Where: • F1A620EAB34F38; Path=/ Set-Cookie: ip_isocode=US Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT;␣ ˓→Path=/ 2011-04-06 14:49:50-0300 [scrapy.core.engine] DEBUG: Crawled (200)

Some of the tips are based on this post from ScrapingHub’s blog [https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/]. Working with relative XPaths Keep in mind that if Selector(text='') >>> sel.css('.shout').xpath('./time/@datetime').getall() ['2014-07-23 19:00'] This is cleaner than using the verbose Whose SHA1 hash is: 3afec3b4765f8f0a07b78f98c07b83f013567a0a Will be downloaded and stored in the following file: /full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg Where:

Selector(text='') >>> sel.css('.shout').xpath('./time/@datetime').getall() ['2014-07-23 19:00'] This is cleaner than using the verbose SHA-1 hash is: 3afec3b4765f8f0a07b78f98c07b83f013567a0a Will be downloaded and stored using your chosen storage method and the following file name: 3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg Custom File F1A620EAB34F38; Path=/ Set-Cookie: ip_isocode=US Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT;␣ ˓→Path=/ 2011-04-06 14:49:50-0300 [scrapy.core.engine] DEBUG: Crawled (200)

with Scrapy selectors, based on this post from ScrapingHub’s blog [https://blog.scrapinghub.com/2014/07/17/xpath-tips- from-the-web-scraping-trenches/]. If you are not much familiar with XPath yet, you Selector(text='') >>> sel.css('.shout').xpath('./time/@datetime').extract() [u'2014-07-23 19:00'] This is cleaner than using the verbose Whose SHA1 hash is: 3afec3b4765f8f0a07b78f98c07b83f013567a0a Will be downloaded and stored in the following file: /full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg Where: